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R-3604/2-AF  A  New  View  of  Weapon  System 
Reliability  and  Maintainability. 

J.  R.  (aebman,  D.  W.  Mclver, 

H.  L.  Shulman.  January  1989. 

This  report  presents  a  new  approach  for 
evaluating  weapon  system  reliability  and 
maintainability  (R&M) .  .  The  approach  is 
based  on  the  frequency  with  which  faults  in 
equipment  degrade  its  ability  to  dependably 
deliver  the  full  measure; of  its  designed 
capabilities,  and  on  the  efficiency  with 
which  maintenance  technicians  remove 
faults,  thereby  restoring  those 
capabilities.'  The  authors  identify 
promising  opportunities  for  strengthening 
policies  and  procedures  that  could  address 
these  weaknesses  a'qd  improve  the  overall 
management  of  R&M.  To  strengthen  the 
support  process,  they  make  the  following 
recommendations:  (1)  debrief  pilot3  for 

all  indications  of  faults,  (2)  track 
performance  of  avionics  by  equipment  serial 
number,  (3)  share  information  about  fault 
symptoms  across  maintenance  levels  (flight 
line  to  shop  to  depot),  and  (4)  establish  a 
special  program  to  repair  problem  unit3  or 
components.  In  addition  to  changes  in  the 
support  process,  the  authors  suggest  some 
adjustments  to  the  product  improvement 
process  and  the  acquisition  process.  (See 
also  R-2908/1,  N-2479,  N-2499,  N-2549.) 
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PREFACE 


This  report  and  its  executive  summary1  integrate  the  final  results  of 
the  project  “Methods  and  Strategies  for  Improving  Weapon  System 
Reliability  and  Maintainability”  conducted  within  RAND’s  Project  AIR 
FORCE  Resource  Management  Program.  This  project,  sponsored  by 
the  Air  Force  Special  Assistant  for  Reliability  and  Maintainability, 
examined  tactical  aircraft  weapon  systems. 


BACKGROUND 

The  Air  Force  Special  Assistant  for  Reliability  and  Maintainability 
and  The  RAND  Corporation  jointly  developed  the  research  plan  that 
called  for  RAND  to  develop  methods  and  strategies  for  improving 
weapon  system  reliability  and  maintainability  (R&M).  The  sponsor 
and  RAND  agreed  that  the  research  should  concentrate  on  tactical  air¬ 
craft  weapon  systems  and  answer  four  questions: 

•  What  kinds  of  payoffs  or  benefits  can  the  Air  Force  expect 
from  improved  R&M? 

•  What  kind  of  information  currently  contained  in  the  Air  Force 
Maintenance  Data  Collection  (MDC)  system  is  useful  in  the 
management  of  R&M? 

•  Are  warranties  an  effective  way  to  achieve  better  R&M? 

•  Can  R&M  be  improved  so  that  present  and  future  U.S.  fighter 
aircraft  can  deliver  their  full  designed  capability  and  maintain 
their  margin  of  superiority  in  the  face  of  a  growing  Soviet 
threat? 

Answers  to  these  questions  and  the  supporting  research  are  docu¬ 
mented  in: 

J.  B.  Abell,  T.  F.  Kirkwood,  R.  L.  Petruschell,  and  G.  K.  Smith, 
The  Cost  and  Performance  Implications  of  Reliability  Improve¬ 
ments  in  the  F-16A/B  Aircraft,  The  RAND  Corporation, 
N-2499-AF,  March  1988. 

R.  L.  Petruschell,  G.  K.  Smith,  and  T.  F.  Kirkwood,  Using  the  Air 
Force  Maintenance  Data  Collection  System  Data  to  Identify 

'R-JWM/l-AF. 
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Candidates  for  Improvement  in  Reliability  and  Maintainability, 
The  RAND  Corporation,  N-2549-AF,  March  1987. 

J.  P.  Stucker  and  G.  K.  Smith,  Warranties  for  Weapons:  Theory 
and  Initial  Assessment,  The  RAND  Corporation,  N-2479-AF, 
April  1987. 

J.  R.  Gebman  and  H.  L.  Shulman,  with  C.  L.  Batten,  A  Strategy  for 
Reforming  Avionics  Acquisition  and  Support,  The  RAND  Corpora¬ 
tion,  R-2908/2-AF,  and  Executive  Summary,  R-2908/1-AF,  July 
1988. 

This  last  effort  involved  RAND’s  participation  in  special  data  collec¬ 
tion  and  analysis  for  the  F-15  C/D  radar  and  the  F-16  A/B  radar.  It 
was  part  of  the  Air  Force’s  special  program  on  F-15/F-16  Radar  R&M 
Improvement.  This  special  program  was  an  outgrowth  of  a  previous 
RAND  project  examining  acquisition  and  support  of  aviation  electron¬ 
ics  (avionics)  equipment. 


SUMMARY 


A  new  approach  for  viewing  weapon  system  reliability  and  main¬ 
tainability  (R&M)  is  developed  to  illuminate  the  dominant  R&M- 
releted  weaknesses  in  the  Air  Force’s  support,  improvement,  and 
acquisition  of  complex  aircraft  weapon  systems.  This  report  identifies 
promising  opportunities  for  strengthening  policies  and  procedures  that 
could  address  these  weaknesses  and  strengthen  the  overall  management 
of  R&M. 


VIEWS  OF  RELIABILITY  AND  MAINTAINABILITY 

In  an  earlier  era  it  may  have  been  appropriate  to  assume: 

•  If  the  operators  don’t  complain  about  it,  it  isn’t  broken. 

•  If  technicians  can’t  duplicate  alleged  symptoms,  it  isn’t  broken. 

•  If  it  isn’t  broken,  don’t  fix  it. 

•  If  it  was  broken,  it’s  fixed  as  soon  as  the  technicians  decide  that 
their  actions  have  corrected  the  problem. 

Although  the  complexities  of  mission  essential  military  equipment  have 
long  defied  such  simple  assumptions,  the  processes  for  acquiring, 
improving,  and  supporting  complex  systems  continue  to  be  driven  by  a 
set  of  measures  that  reinforces  such  a  traditional  view  of  R&M: 

•  Fully  mission  capable  (FMC)  rate:  a  measure  of  availability 
influenced  largely  by  the  pilot’s  subjective  assessment  of 
whether  maintenance  is  needed; 

•  Mean  time  between  failure  (MTBF):  a  measure  of  reliability 
influenced  mostly  by  whether  technicians  in  the  air  base’s  shop 
execute  a  repair  action; 

•  Mean  time  to  repair  (MTTR):  a  measure  of  how  quickly  the 
technicians  on  the  flight  line  complete  their  work. 

For  highly  visible  failures,  such  as  total  failure  of  a  major  subsystem, 
such  measures  have  been  very  meaningful.  Meaningfulness  breaks 
down,  however,  as  the  visible  symptoms  of  failure  become  more 
obscure.  With  the  continuing  progress  in  the  reliability  of  Air  Force 
equipment,  total  failure  is  becoming  increasingly  rare.  Today,  the 
dominant  problem  with  most  equipment  is  not  in  totally  lost  perfor¬ 
mance,  but  is  in  the  form  of  degraded  performance. 


VI 


For  complex  military  equipment  that  provides  combat-essential 
functions,  we  believe  that  the  primary  objective  of  R&M  should  be 
dependable  delivery  of  the  full  measure  of  capabilities  that  the  equip¬ 
ment  is  designed  to  deliver  during  its  operational  life. 

Currently,  the  main  threat  to  this  objective  comes  from  the  type  of 
performance -eroding  fault  that  manifests  symptoms  only  in  particular 
situations.  Faults  with  such  nonstationary  observability  are 
what  we  term  Type  B  faults.  In  contrast,  a  Type  A  fault  is  one 
where  symptoms  are  observable  no  matter  when  or  where  the  faulty 
item  of  equipment  is  operated  or  tested.  Faults  with  such  stationary 
observability  have  the  kind  of  visibility  that  is  needed  for  the  tradi¬ 
tional  measures  of  R&M  to  be  meaningful.  However,  equipments  fre¬ 
quently  afflicted  with  Type  B  faults  require  a  new  approach. 

The  new  approach  must  deal  with  Type  A  and  Type  B  faults  and  it 
must  provide  full  visibility  of  the  phenomena  that  determine  R&M: 

•  Fault  initiation, 

•  Fault  removal. 

To  do  this,  we  propose  a  view  of  R&M  based  upon: 

•  The  frequency  with  which  new  faults  initiate  within 
equipment  degrade  the  equipment’s  ability  to  dependably 
deliver  the  full  measure  of  its  designed  capabilities. 

•  The  efficiency  with  which  maintenance  technicians 
remove  faults,  thereby  fully  restoring  designed  capabilities. 

To  apply  such  considerations  to  an  assessment  of  the  R&M  situation 
for  a  particular  subsystem,  the  Air  Force  must  be  able  to  detect  degra¬ 
dations  in  the  performance  of  subsystems.  In  many  instances,  how¬ 
ever,  the  Air  Force  lacks  a  direct  and  definitive  capability  to  do  this  in 
the  environment  of  routine  operations.  In  such  situations  it  must  rely 
on  indications  gleaned  from  both  pilot  observations  and  symptoms 
detected  by  built-in  tests  (BIT).  Although  an  indication  of  diffi¬ 
culty  from  a  single  flight  may  not  provide  satisfactory  evidence 
of  a  fault,  it  nonetheless  needs  to  be  documented  and  inter¬ 
preted  in  the  context  of  any  related  indications  so  important 
patterns  may  come  to  be  recognized. 


THE  SUPPORT  PROCESS 
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The  weapon  system  support  process  hinders  technicians  when  they 
attempt  to  solve  the  extraordinarily  difficult  challenges  presented  by 
Type  B  faults  because  the  current  process: 

•  Provides  too  little  information  about  avionics  equipment  perfor¬ 
mance  during  routine  training  flights, 

•  Fails  to  track  avionics  equipment  performance  by  serial 
number, 

•  Inadequately  integrates  maintenance  efforts  and  fault  informa¬ 
tion  across  maintenance  levels, 

•  Has  inadequate  capabilities  to  fix  bad  actor  equipment. 

To  strengthen  the  support  process  we  recommend  debriefing  pilots 
for  all  indications  of  faults,  tracking  performance  of  avionics  by  equip¬ 
ment  serial  number,  sharing  meaningful  information  about  fault  symp¬ 
toms  across  maintenance  levels  (flight  line  to  shop  to  depot),  and 
establishing  a  special  program  to  repair  bad  actor  equipment. 

Debrief  Pilots  for  All  Indications  of  Faults 

Pilots  need  to  share  with  maintenance  all  indications  of  difficulties 
that  they  and  the  BIT  detect  in  the  operation  of  complex  weapon  sys¬ 
tems.  They  do  not  necessarily  have  to  request  maintenance  every  time 
they  notice  an  indication,  but  they  need  to  help  maintenance  track  the 
performance  history  of  complex  subsystems. 

Track  Performance  of  Avionics  by  Equipment 
Serial  Number 

The  performance-oriented  tracking  of  complex  subsystems  that 
begins  with  pilot  debriefs  needs  to  continue  with  careful  tracking  of  the 
subsystem’s  major  components  (both  LRUs  and  SRUs),  each  of  which 
has  a  unique  serial  number.  Maintenance  needs  to  keep  accurate  and 
updated  records  of  which  units  are  being  circulated  between  the  air¬ 
craft  on  the  flight  line  and  the  shop  and  depot.  This  step  is  essential 
in  helping  the  shop  and  depot  track  and  identify  the  bad  actors  that 
are  in  greatest  need  of  special  attention. 
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Share  Information  About  Fault  Symptoms 
Across  Maintenance  Levels 

Because  the  support  process  must  use  different  tests  and  different 
pass/fail  criteria  at  each  maintenance  level,  sharing  of  meaningful 
information  is  especially  important  to  verify  that  a  maintenance  action 
has  addressed  the  same  fault  that  was  detected  at  the  previous  mainte¬ 
nance  level.  Such  verification  is  especially  needed  when  one  is  trying 
to  rectify  Type  B  faults.  One  beneficial  way  to  improve  information 
sharing  would  be  to  provide  test  translation  dictionaries  that  would 
enable  avionics  technicians  at  one  maintenance  level  to  translate  test 
results  from  another  level  into  terms  they  would  find  useful  for  iden¬ 
tifying  bad  actors. 

Fix  the  Bad  Actors 

The  next  key  element  of  the  strategy  for  improving  the  support  pro¬ 
cess  is  to  repair  the  problem  units  or  components  so  that  they  do  not 
circulate  between  the  support  process  and  airplanes  in  degraded  condi¬ 
tion.  To  improve  the  efficiency  of  repairing  bad  actors,  we  recommend 
improved  fault-isolation  capabilities: 

•  Direct  entry  into  test  sequences  for  specific  sections  of  lengthy 
ground  avionics  tests  at  both  the  base  and  depot, 

•  Loop  testing  for  specific  tests  at  both  the  base  and  depot, 

•  Special  en  ^ironmental  and  system  bench  capabilities  for  depots. 


THE  PRODUCT  IMPROVEMENT  PROCESS 

The  product  improvement  process  can  improve  the  reliability  of  the 
airborne  equipment  and  increase  the  capability  of  the  support  process 
to  remove  faults  efficiently.  However,  the  process  is  limited  as  a  tool 
for  improving  R&M  because  it 

•  Functions  with  inadequate  information  about  dominant  R&M 
problems, 

•  Fails  to  implement  important  improvements  in  a  timely  way. 

To  strengthen  the  responsiveness  of  the  prodrn  <.  improvement  pro¬ 
cess  to  the  root  causes  of  a  weapon  system’s  dominant  R&M  problems, 
we  propose  a  strategy  that  includes  improving  the  flow  of  information 
from  the  field,  increasing  field  engineering,  and  expediting  improve¬ 
ments. 


Improve  Information  from  the  Field 

By  applying  the  proposed  new  view  nf  R&M  to  data  already  being 
acquired  by  the  Air  Force’s  existing  maintenance  data  collection  sys¬ 
tem,  the  Air  Force  can  better  identify  areas  where  further  R&M  inves¬ 
tigation  is  most  needed.1  Further  improvement  can  come  from  monitor¬ 
ing  and  analyzing  information  from  the  previously  proposed  tracking  of 
avionics  performance  by  equipment  serial  number.  Monitoring  and 
analyzing  problematic  equipment  and  the  reasons  why  the  routine  sup¬ 
port  process  fails  to  correct  their  problems  would  make  the  product 
improvement  process  more  aware  of  the  dominant  problems  that  are 
undermining  fault  isolation  capabilities. 

Increase  Field  Engineering 

After  the  initial  fielding  of  a  weapon  system,  additional  engineering 
is  required  for  any  system — especially  sophisticated  systems — to  raise 
the  maturity  of  its  R&M  characteristics  to  a  suitably  acceptable  level. 
One  way  to  do  this  is  to  get  equipment  contractor  engineers  more 
involved  in  understanding  R&M  from  the  viewpoint  of  the  operators 
and  the  maintainers  in  the  field.  Although  a  capability  to  do 
performance-oriented  tracking  of  equipment  (especially  after  attempted 
repairs)  will  help,  an  actual  presence  in  the  field  will  be  required  to 
identify  the  root  causes  of  dominant  problems  in  the  more  complex 
subsystems. 

Exnedite  Important  Improvements 

Problems  could  be  identified  earlier  if  acquisition  programs  routinely 
included  more  detailed  data  collection  efforts  by  the  contractors  for  the 
more  complex  subsystems.  If  the  Air  Force  developed  special  pro¬ 
cedures  that  would  expedite  management  and  would  preprovision 
funds,  critical  improvements  would  be  accelerated. 

THE  ACQUISITION  PROCESS 

The  acquisition  process  represents  the  first  line  of  defense  for  R&M. 
Weapor  systems  with  better  R&M  characteristics  would  lessen  the 
need  for  product  improvement  and  lighten  the  support  burden.  How- 

'R.  L.  /etruschell,  G.  K.  Smith,  and  T.  F.  Kirkwood,  Using  the  Air  Force  Mainte¬ 
nance  Data  Collection  System  Data  to  Identify  Candidates  for  Improvement. 
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ever,  the  effectiveness  of  the  acquisition  process,  in  terms  of  R&M,  is 
limited  because  it 

•  Fails  to  use  a  meaningful  set  of  management  measures  for 
R&M, 

•  Lacks  a  process  for  setting  rationally  based  R&M  goals, 

•  Does  not  assure  delivery  of  needed  levels  of  R&M. 

To  strengthen  the  role  of  the  acquisition  process  in  managing  ^ 

weapon  system  R&M,  we  recommend  expanding  awareness  of  R&M 
deficiencies,  emphasizing  fault  removal  efficiency,  accelerating  matura¬ 
tion  of  avionics,  and  reorganizing  avionics  development. 

Expand  Awareness  of  R&M  Deficiencies.  To  improve  the  R&M 
record  of  the  acquisition  process,  the  government  and  industry  organi¬ 
zations  that  are  responsible  for  the  development  of  new  equipment 

need  to  expand  their  awareness  of  dominant  R&M  deficiencies.  i 

Emphasize  Fault  Removal  Efficiency.  Improved  awareness  of  ' 

the  dominant  problems  is  only  an  initial  step  that  needs  to  be  followed 
by  a  greater  emphasis  on  fault  removal  efficiency  in  new  developments. 

Accelerate  Maturation  of  Avionics.  This  element  of  the  strategy 
for  strengthening  the  acquisition  process  aims  at  more  timely  and  ' 

fuller  achievement  of  R&M  goals.  Although  it  emphasizes  aviation  / 

electronics  (avionics),  because  this  class  of  equipment  currently  f 

presents  the  greatest  R&M  challenges,  the  concept  of  maturation  is 

applicable  to  the  development  of  very  complex  systems.  I 

Our  general  concept  views  the  research  and  development  of  complex  I 

weapon  systems  as  a  process  that  has  six  basic  phases:  i 

< 

•  Technology  development 

•  Critical  component  development 

•  Subassembly  development 

•  Assembly /unit  development 

•  Subsystem  development  ( 

•  Weapon  system2  integration  development. 

An  orderly  and  efficient  development  program  will  invest  just  the  right 
amount  of  time  and  resources  in  each  phase;  and  although  phases  will 
overlap,  they  will  be  neither  initiated  nor  terminated  too  early.  During 
each  basic  phase,  we  use  the  concept  of  maturity  as  a  qualitative  gauge 

of  the  status  of  development  efforts.  j 

Whether  one  is  initiating  or  terminating  a  phase,  the  decision  should  ( 

be  based  on  scientifically  accumulated  evidence  of  progress  and  an  j 


including  the  ground  support  equipment  peculiar  to  that  weapon  system. 
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objective  assessment  of  the  likelihood  that  lingering  difficulties  can  be 
resolved  before  the  next  phase  gets  too  far  along. 

Throughout  the  development  process,  there  is  pressure  to  initiate 
the  next  basic  phase  sooner  rather  than  later.  Even  when  done  too 
soon,  development  programs  often  survive,  although  R&M  characteris¬ 
tics  may  suffer.  Coping  with  the  pressures  during  the  latter  phases  has 
been  the  chief  consideration  of  RAND’s  recent  avionics  research,  which 
has  yielded  a  proposal  that  is  a  form  of  the  general  concept  of  matura¬ 
tion  that  we  call  maturational  development.3  To  implement  this  con¬ 
cept  for  subsystem  and  weapon  system  integration  development,  the 
Air  Force  needs  a  formal  period  in  the  acquisition  process  in  which 
development  programs  are  required  to  set  aside  time  and  resources  for: 

•  Measuring  operational  experience,  organizing  and  recording 
R&M-related  data,  interpreting  the  data,  and  drawing  conclu¬ 
sions  about  the  causes  of  the  problems  that  are  responsible  for 
any  R&M  shortfalls. 

•  Correcting  R&M  deficiencies  before  transfer  of  program 
management  responsibility  to  the  Air  Force  Logistics  Com¬ 
mand. 

We  see  a  formal  maturational  development  phase  as  most  beneficial 
for  three  classes  of  complex  combat-essential  avionics  subsystems: 

•  New  subsystems  that  are  just  beginning  development, 

•  Already  fielded  subsystems  that  we  being  modified  to  improve 
their  functional  performance, 

•  Already  fielded  subsystems  where  improvements  in  R&M  would 
substantially  narrow  the  gap  between  designed  and  opera¬ 
tionally  available  performance. 

The  cost  to  retrofit  R&M  improvements  can  be  quite  high.  Matura¬ 
tional  development  offers  the  largest  benefit-to-cost  ratio  when  aimed 
at  new  avionics  subsystems  that  are  just  beginning  development  (Phase 
V  of  the  development  of  a  weapon  system).  In  such  cases,  it  should 
occur  before  high-rate  production  to  avoid  high  retrofit  costs.4 

Reorganize  Avionics  Development.  This  final  element  aims  at 
reducing  the  R&M-related  development  problems  that  occur 

3Gebtnan,  Shulman,  and  Batten,  1988. 

Approaches  that  would  provide  the  time  to  incorporate  a  maturational  development 
phase  include:  (1)  defer  the  onset  of  high  rate  production,  and  (2)  start  full-scale 
engineering  development  early.  The  Air  Force’s  System  Program  Office  for  the 
Advanced  Tactical  Fighter  is  working  towards  implementing  a  combination  of  such 
approaches. 
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throughout  weapon  system  development.  We  propose  a  reorganization 
of  avionics  development  responsibilities  that  has  two  goals: 

•  Expediting  the  maturation  and  application  of  new  technologies 
by  improving  government  and  industry  sponsored  R&D  during 
critical  component  development,  subassembly  development,  and 
assembly/unit  development.5 

•  Institutionalizing  maturational  development  during  subsystem 
and  weapon  system  integration  development. 

Ideally,  subsystem  development  (Phase  V)  would  start  far  enough  ahead 
of  weapon  system  integration  development  (Phase  VI)  to  allow  a  matura¬ 
tional  development  effort  to  be  underway  before  Phase  VI  begins. 
Although  a  Phase  V  application  of  maturational  development  would 
require  hosting  the  subsystem  on  a  different  weapon  system  for  the  gath¬ 
ering  of  operational  experience,  the  advantage  of  early  development  of 
critical  subsystems  is  that  design  improvements  can  be  incorporated 
before  high  rate  production  starts  for  the  new  host  weapon  system. 

Such  reorganization  and  the  attendant  increased  role  for  the  govern¬ 
ment  in  avionics  development  will  increase  the  cost  of  acquiring  avion¬ 
ics  equipment,  at  least  initially.  Moreover,  elements  of  the  overall 
strategy  for  strengthening  the  support  process,  the  product  improve¬ 
ment  process,  and  other  aspects  of  the  acquisition  will  also  increase 
costs.  However,  the  extent  of  weapon  system  R&M  improvement  will 
determine  reduction  in  total  lifecycle  costs.  Moreover,  improved  R&M 
management  will  increase  the  readiness  of  equipment  to  deliver  the  full 
measure  of  capabilities  that  it  is  designed  to  deliver,  especially  in  com¬ 
bat. 


5See  Gebman,  ShuJman,  and  Batten,  1988,  for  details. 
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I.  INTRODUCTION 


Although  the  Air  Force  and  industry  deserve  much  credit  for 
improving  reliability  and  maintainability  (R&M)  in  aircraft  weapon 
systems,  there  is  a  major  gap  in  the  current  emphasis,  a  consequence  of 
reliance  on  measures  that  display  only  an  incomplete  picture  of  the 
R&M  landscape.  Important  missing  situations  are  problems  that 
weaken  weapon  systems’  ability  to  dependably  deliver  all  of  their 
designed  performance  that  could  be  essential  in  combat.  These  missing 
characteristics  include  situations  in  which: 

•  The  pilot  sees  a  symptom  of  a  fault  but  does  not  report  it  to 
maintenance. 

•  The  pilot  reports  a  symptom  as  a  discrepancy  deserving  the 
attention  of  maintenance  technicians,  but  the  flight  line  techni¬ 
cians  do  not  remove  a  line  replaceable  unit  (LRU)  from  the  air¬ 
plane  because  they  could  not  duplicate  (CND)  symptoms  of  the 
fault. 

•  The  airplane’s  Built-In-Test  (BIT)  detects  a  fault  during  a 
flight,  but  flight  line  technicians  do  not  remove  an  LRU  from 
the  aircraft  because  they  reran  the  BIT  test  and  it  failed  to 
detect  the  fault,  and  the  technicians  could  not  duplicate  symp¬ 
toms  of  the  fault. 

•  Flight  line  technicians  remove  an  LRU  from  an  airplane  and 
send  it  to  the  avionics  shop,  but  the  shop  technicians  find  that 
the  LRU  bench  checks  serviceable  (BCS)  when  they  test  it 
because  their  tests  failed  to  detect  symptoms  of  the  fault. 

•  Shop  technicians  remove  a  shop  replaceable  unit  (SRU)  from 
an  LRU  and  send  it  to  the  depot  repair  center,  but  the  depot 
technicians  find  that  the  SRU  retests  OK  (RETOK)  when  they 
test  it  because  their  tests  failed  to  detect  symptoms  of  the  fault. 

By  failing  to  reflect  such  situations,  the  traditional  view  of  R&M  limits 
management’s  awareness  of  the  overall  R&M  picture.  Such  partial 
awareness  results  in  insufficient  capabilities  to  fully  support  the 
designed  capabilities  of  mission-essential  equipment. 

Although  the  complexities  of  mission-essential  military  equipment 
have  long  defied  the  simplifying  assumptions  upon  which  the  tradi¬ 
tional  view  has  stood,  the  processes  for  acquiring,  improving,  and  sup- 
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porting  complex  systems  continue  to  be  driven  by  a  set  of  measures1 
that  reinforces  a  traditional  view  of  R&M.  For  highly  visible 
failures — for  example,  total  failure  of  a  major  subsystem — such  mea¬ 
sures  have  been  very  meaningful.  However,  the  complexities  of  modern 
equipment  have  caused  visible  symptoms  of  failure  to  become  more 
obscure. 


DATA  SOURCES 

This  research  is  based  on  detailed  examinations  of  one  particular 
aircraft  weapon  system,  the  F-16  A/B,  and  on  in-depth  examinations  of 
two  critical  radars,  the  APG-66  on  the  F-16  A/B  and  the  APG-63  on 
the  F-15  C/D.  Although  this  study  uses  data  currently  available  from 
the  Air  Force  MDC  system,  important  aspects  of  the  new  view  of  R&M 
are  more  fully  illustrated  by  data  that  are  available  at  present  only  by 
special  means.  Such  data  were  collected  as  part  of  the  F-15/F-16 
Radar  R&M  Improvement  Program: 

•  Data  on  the  F-16’s  APG  66  radar  were  collected  from  150  F-16s 
monitored  during  a  six-month  period  (June  to  December  1984) 
at  Hill  AFB  and  Hahn  Air  Base  and  covered  16,077  flights  and 
the  resulting  maintenance  at  these  bases  and  the  depot. 

•  Data  on  the  F-15’s  APG  63  radar  were  collected  from  150  F-15s 
monitored  during  the  same  period  at  Langley  AFB  and  Bitburg 
Air  Base  and  covered  16,702  flights  and  the  resulting  radar 
maintenance  at  these  bases  and  depot. 

Contractor  personnel  interviewed  pilots  after  maintenance  debrief, 
documented  all  symptoms  of  faults  observed  by  the  pilot — including 
BIT  detected  symptoms — and  documented  maintenance  on  the  flight 
line,  in  the  shop,  and  at  the  depot.  To  collect  these  data,  the  radar 
contractors2  and  the  weapon  system  prime  contractors3  deployed  72 
people  to  the  four  air  bases  and  the  two  depots  for  six  months. 

The  Air  Force  conducted  the  F-15/F-16  Radar  R&M  Improvement 
Program  in  response  to  a  recommendation  from  a  RAND  project  that 
the  commander  of  the  Air  Force  Systems  Command4  had  requested  to 
analyze  ways  of  improving  avionics  acquisition  and  support.  That  proj- 


‘See  Sec.  II. 

2Hughes  Aircraft  Radar  Systems  Group  for  the  APG  63  and  Westinghouse  Defense 
Electronics  Center  for  the  APG  66. 

3McDonnelI  Aircraft  Company  for  the  F-15  and  General  Dynamics  for  the  F-16. 

4Who  then  was  General  Alton  D.  Slay. 
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ect  also  helped  the  Air  Force  establish  the  data  collection  and  analysis 
phase  of  that  program.5 

By  continuing  to  assist  that  special  Air  Force  program,  the  present 
project  has  been  able  to  acquire  crucial  data  and  to  explore  specific 
means  for  implementing  several  methods  described  in  this  report. 
Such  research  has  examined  methods  where  implementation  is  most 
likely  to  be  complicated  by  the  need  to  alter  long-established  policies 
and  procedures. 


ORGANIZATION6 

To  help  close  the  gap  in  the  current  emphasis  on  R&M,  and  to  lay  a 
basis  for  developing  methods  and  a  strategy  for  improving  weapon  sys¬ 
tem  R&M,  Sec.  II  presents  a  new  approach  to  viewing  R&M,  which  is 
used  in  subsequent  sections  to  analyze  the  Air  Force’s  management  of 
weapon  system  R&M.  Weaknesses  and  corresponding  opportunities 
for  improvement  are  found  in  three  areas:  the  support  process  (Sec. 
Ill),  the  product  improvement  process  (Sec.  IV),  and  the  acquisition 
process  (Sec.  V).7  A  comprehensive  strategy  is  formed  to  strengthen 
the  Air  Force’s  ability  to  manage  ‘be  R&M  of  its  aircraft  weapon  sys¬ 
tems.  Section  VI  summarizes  the  recommended  strategy  and  states  our 
conclusions. 


6Gebman,  Shulman,  and  Batten,  1988. 

'The  content  and  organization  of  this  report  closely  parallel  the  project’s  final  sum¬ 
mary  briefing  presented  to  Air  Force  audiences  during  1987. 

7We  take  up  the  support  process  first  because  opportunities  for  improving  it  are 
important  to  strengthening  product  improvement  and  the  acquisition  process.  Second, 
we  take  up  the  product  improvement  process  because  opportunities  there  are  important 
to  strengthening  the  acquisition  process. 


II.  VIEWS  OF  RELIABILITY  AND 
MAINTAINABILITY 


The  traditional  view  of  R&M  masks  problems  that  weaken  weapon 
systems’  ability  to  deliver  the  full  measure  of  their  designed  perfor¬ 
mance.  To  help  assure  the  dependable  delivery  of  mission-essential 
performance,  we  present  a  new  approach  to  viewing  R&M,  describing 
why  such  an  approach  is  needed,  comparing  the  new  view  with  the  tra¬ 
ditional  view,  and  providing  an  example  application.  The  example 
illustrates  how  the  traditional  incomplete  picture  can  mask  problems  of 
potentially  major  consequence  to  the  combat  effectiveness  of  mission- 
essential  equipment. 


THE  TRADITIONAL  VIEW 

The  traditional  view  is  a  carryover  from  an  earlier  era  when  it  may 
have  been  appropriate  to  assume: 

•  If  the  operators  don’t  complain  about  it,  it  isn’t  broken. 

•  Even  if  they  complain,  if  the  technicians  can’t  duplicate  the 
alleged  symptom,  it  isn’t  broken.1 

•  If  it  isn’t  broken,  don’t  fix  it. 

•  If  it  was  broken,  it’s  fixed  as  soon  as  the  technicians  decide  that 
their  actions  have  corrected  the  problem.2 

This  simple  view  assumes  that  the  equipment  is  either  broken  or  fixed 
and  both  conditions  are  easy  to  identify.  Such  a  view  leads  to  simple 
measures  for  characterizing  R&M: 

•  Fully  mission  capable  (FMC)  rate:  a  measure  of  availability 
influenced  largely  by  the  pilot’s  subjective  assessment  of 
whether  maintenance  is  needed; 

•  Mean  time  between  failure  (MTBF):  a  measure  of  reliability 
influenced  mostly  by  whether  technicians  in  the  air  base’s  shop 
find  it  appropriate  to  execute  a  repair  action; 

'Unless  the  reports  persist,  whereupon  techni:ians  will  assume  that  something  is  bro¬ 
ken. 

^The  technicians’  judgment  is  questioned  only  if  the  operators  (flight  crews)  request 
maintenance  after  one  of  the  next  three  flights.  Thus,  if  flight  crews  tolerate  signs  of 
degraded  performance  for  more  than  three  flights  following  a  maintenance  action,  the 
current  system  implicitly  assumes  that  the  maintenance  action  was  fully  effective. 
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•  Mean  time  to  repair  (MTTR):  a  measure  of  how  quickly  the 
technicians  on  the  flight  line  complete  their  work. 

Within  the  simple  traditional  view,  this  set  of  measures  accounts  for 
all  possible  conditions  of  the  equipment.  Unfortunately,  the  world  of 
modern  integrated  avionics  is  not  so  simple.  Some  problems  limit  abil¬ 
ity  to  detect  symptoms  of  serious  faults;  and  even  when  detected,  some 
problems  limit  responses  to  such  symptoms.  These  problems  are 
masked  by  the  limited  vision  of  the  traditional  view. 

Limited  Capabilities  to  Detect  Fault  Symptoms 

When  equipment  fails  to  deliver  its  full  measure  of  designed  perfor¬ 
mance,  the  underlying  fault  often  causes  a  degradation  in  performance 
that  is  subtle  and  difficult  to  observe. 

Many  factors  make  it  difficult  for  pilots  and  BITs  to  detect  faults  in 
modern  avionics: 

•  Limited  opportunities  to  exercise  certain  subsystems, 

•  Rare  occurrences  of  total  failures, 

•  Varying  degrees  of  degradation, 

•  Intermittently  observable  symptoms, 

•  Incomplete  and  imperfect  testing, 

•  Pilot  workload. 

Limited  Opportunities  to  Exercise  Certain  Subsystems.  Rou¬ 
tine  peacetime  missions  provide  limited  opportunities  to  exercise  the 
full  capabilities  of  certain  combat-oriented  subsystems,  such  as  fire 
control  radars,  air-to-air  weapon  delivery  systems,  and  electronic  coun¬ 
termeasures  (ECM)  systems. 

Rare  Occurrences  of  Total  Failures.  Nowadays,  reliability  and 
versatility  of  equipment  has  grown  to  the  point  where  equipment  very 
rarely  experiences  total  failure.  Rather,  it  falls  victim  to  faults  that 
erode  its  performance  superiority  over  potential  enemy  weapons. 
Indeed,  graceful  degradation  of  performance  is  the  mode  of  failure  that 
developers  strive  to  achieve  in  their  designs.  However,  that  can  be  dif¬ 
ficult  for  pilots  to  detect,  especially  when  they  cannot  judge  what  level 
of  performance  the  equipment  was  designed  to  deliver.3 

Varying  Degrees  of  Degrad  ion.  To  further  complicate 
matters,  the  degree  of  degradation  nay  vary  across  the  different 

3A  fire  control  radar  is  a  classic  example  of  equipment  whose  designed  performance 
depends  upon  the  situation  of  use.  Target  detection,  for  example,  will  occur  at  different 
ranges  depending  upon  the  target,  its  orientation  relative  to  the  radar,  and  the  back¬ 
ground  clutter. 
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functions  that  a  subsystem  performs,  with  some  functions  rarely  being 
executed  during  peacetime  training  missions.  On  such  missions  a  pilot 
may  see  no  symptoms  whatsoever  for  combat-critical  faults.  Fortu¬ 
nately,  modem  subsystems  have  built-in  tests  that  can  monitor  equip¬ 
ment  performance  and  often  detect  symptoms  of  such  faults.  However, 
false  alarms  and  other  problems  with  some  BIT  systems,  especially  the 
early  ones  that  were  developed,  have  given  BIT  a  spotty  reputation 
that  undermines  its  credibility  with  pilots  and  maintenance  techni¬ 
cians. 

Intermittently  Observable  Symptoms.  The  symptoms  of 
degraded  or  failed  performance  may  appear  intermittently.  Faults  in 
sophisticated  subsystems  exhibit  different  symptoms  depending  on  the 
operating  mode  and  environment.  In  some,  the  equipment  may  func¬ 
tion  properly;  in  others,  it  may  deliver  only  part  of  its  designed  perfor¬ 
mance;  in  yet  others,  it  may  totally  fail. 

Some  faults  manifest  different  symptoms  in  different  operating 
environments.  Equipments  that  rely  on  sensors  (such  as  the  antenna 
in  a  radar)  for  their  primary  source  of  information  are  designed  to  per¬ 
form  more  or  less  well  depending  on  the  background  environment  in 
which  the  object  is  sensed  and  on  the  object’s  movement  within  that 
environment.  If  a  fault  has  degraded  the  equipment’s  performance 
capabilities,  it  may  fail  to  detect  a  target  in  a  highly  cluttered  environ¬ 
ment,  whereas  it  might  do  fine  in  an  uncluttered  environment. 

The  operating  environment  can  have  another  effect  as  well.  Symp¬ 
toms  of  some  faults  (such  as  mechanical  flaws  in  connections)  are  trig¬ 
gered  by  such  environmental  changes  as  temperature,  vibration,  and 
deflection  under  high  flight  loads  (e.g.,  pulling  nine  gs).  A  loosely  sol¬ 
dered  wire  may  manifest  symptoms  that  the  pilot  can  observe  only 
while  the  aircraft  is  executing  a  high  g  maneuver.  (Worse,  mainte¬ 
nance  technicians  may  not  be  able  to  observe  its  symptoms  because  of 
lack  of  an  environmental  test  chamber.)  To  further  complicate 
matters,  the  pilot  may  not  know  or  remember  all  environmental  condi¬ 
tions  that  may  have  influenced  the  manifestation  of  the  fault’s  symp¬ 
toms. 

Testing  Is  Incomplete  and  Imperfect.  Because  of  problems  with 
relying  only  on  pilot  observation  and  judgment,  maintenance  techni¬ 
cians  are  depending  more  and  more  on  BITs  of  circuits,  not  only  in  fire 
control  radars  but  also  in  such  subsystems  as  Electronic  Countermea¬ 
sure  Equipment  in  which  certain  modes  are  used  irregularly.  Though 
technicians  are  becoming  critically  dependent  on  these  tests  of  circuits, 
the  tests  have  limitations:  They  are  not  comprehensive,  many  of  them 
are  not  continuous,  and  most  do  not  indicate  the  severity  of  a  problem. 
Moreover,  the  symptoms  of  many  faults  occur  intermittently  or  only  in 
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certain  situations  or  environments,  so  the  BITs  can  yield  irregular 
indications  of  serious  faults. 

Neither  BIT  indications  nor  pilot  observations  provide  the  kind  of 
accurate  information  that  maintenance  should  have,  but  both  together 
are  better  than  either  alone.  Consequently,  during  peacetime  it  is 
necessary  to  consider  all  indications  of  problems,  whether  they  be  pilot 
observations  or  BIT  indications. 

Pilot  Work  Load.  Although  the  pilot  directly  controls  some  func¬ 
tions  by  activating  switches  on  the  control  panel,  the  throttle,  and  the 
stick,  computers  directly  control  other  functions.  In  addition,  the  pilot 
often  rapidly  executes  a  series  of  functions  (such  as  search,  acquire, 
track,  weapon  release)  while  coping  with  a  high  workload  and — in  the 
case  of  fighter  pilots — a  physically  exhausting  series  of  maneuvers. 
Thus  when  symptoms  of  a  fault  arise,  the  pilot  may  not  know  the  pre¬ 
cise  operating  mode.  Even  if  he  does,  he  probably  cannot  subsequently 
identify  the  precise  settings  of  all  switches  at  the  time  the  symptoms 
were  observed. 

Fire  control  systems  offer  further  examples  of  equipment  that 
presents  special  difficulties  for  maintenance  technicians  in  detecting 
faults.  For  instance,  pilots  have  no  direct  means  of  assessing  whether 
the  fire  control  radar  is  delivering  the  full  range  of  its  capability. 
Pilots  do  not  use  all  of  the  radar’s  functions  in  every  mission.  The 
performance  they  do  see  is  also  a  function  of  many  factors,  including 
where  the  pilot  is  looking  and  what  he  is  looking  at. 

Limited  Responses  to  Fault  Symptoms 

Uncertainty  about  the  extent  of  performance  degradations  causes 
pilots  and  maintenance  technicians  to  have  problems  determining  their 
responses  to  symptoms  of  a  fault.  If  a  pilot  lacks  strong  corroborating 
evidence  that  a  subsystem  is  broken,  he  is  reluctant  to  tell  mainte¬ 
nance  about  BIT  detected  symptoms.  He  might  also  hesitate  to  share 
information  about  pilot-perceived  symptoms  unless  there  are  strong 
signs  that  the  equipment  is  broken.  Such  practices  are  part  of  a  pilot 
tradition  that  you  don’t  ask  maintenance  technicians  to  fix  something 
unless  you  are  sure  that  it  is  broken. 

Similarly,  technicians  have  been  forced  to  adopt  a  maintenance  tra¬ 
dition  that  you  don’t  replace  an  item  of  equipment  unless  you  are  sure 
that  it  is  broken.  Technicians  become  certain  that  an  item  of  equip¬ 
ment  is  broken  only  if  it  fails  to  pass  one  of  the  tests  that  they  apply 
to  it.  For  example,  if  the  pilot  reports  that  the  equipment  failed  a  BIT 
during  a  flight,  the  flight  line  technicians  will  run  the  BIT  again  on  the 
ground  to  try  to  duplicate  the  BIT-detected  symptom.  If  they  can  not 
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duplicate  (CND)  a  symptom  on  the  flight  line,  or  detect  any  other 
symptom(s)  they  usually  will  not  replace  any  equipment.  Likewise, 
shop  technicians  at  an  air  base  will  not  replace  parts  in  a  unit  they  are 
testing  if  the  unit  bench  checks  serviceable  (BCS);  that  is,  no  symp¬ 
toms  of  a  fault  were  found.  And  at  a  depot,  technicians  will  not 
replace  parts  in  a  unit  if  it  retests  OK  (RETOK)  when  they  run  their 
tests. 

Both  the  pilot  and  maintenance  traditions  have  reasonable  founda¬ 
tions.  First,  spare  parts  often  are  expensive  and  not  always  readily 
available.  Second,  even  in  the  best  of  circumstances,  the  process  of 
replacing  an  item  of  equipment  may  induce  damage  that  is  far  worse 
than  the  original  problem.  These  realities  and  traditions  create  pres¬ 
sures  not  only  to  discount  but  to  dismiss  information  that  indicates  a 
fault  whenever  subsequent  observations  or  tests  fail  to  find  symptoms. 

Modern  equipment,  however,  have  been  plagued  by  high  CND,  BCS, 
and  RETOK  rates,  even  though  the  equipment  often  has  otherwise 
excellent  scores  in  terms  of  the  traditional  measures  of  R&M:  FMC, 
MTBF,  and  MTTR.4  Although  high  rates  for  CND,  BCS,  and  RETOK 
reflect  much  fruitless  maintenance  activity,  the  more  serious  concern  is 
the  combat  preparedness  of  weapon  systems  that  carry  hard  to  fix 
faults  for  prolonged  periods. 


THE  NEW  VIEW 

For  complex  military  equipment  that  provides  combat-essential 
functions,  we  believe  that  the  primary  objective  of  R&M  needs  to  be 
the  full  and  dependable  delivery  of  the  equipment’s  designed  capabili¬ 
ties.5 

Currently,  the  main  threat  to  this  objective  comes  from  the  type  of 
performance-eroding  fault  that  manifests  symptoms  only  in  certain 
situations.  Faults  with  such  nonstationary  observability  are  what  we 

4Gebman,  Shulman,  and  Batten,  1988.  Unfortunately,  faults  have  lingered  in  equip¬ 
ment  for  weeks  and  even  months  before  finally  being  isolated  and  corrected.  The 
F-15/F-16  Radar  R&M  Improvement  Program  documented  this  phenomenon  with  both 
the  APG  63  and  the  APG  66. 

®A  contrary  view  holds  that  it  is  unreasonable  to  expect  equipment  to  continue 
delivering  designed  levels  of  performance  long  after  it  has  been  introduced  into  service. 
However,  the  whole  support  process  (including  BIT  and  tests  at  air  bases  and  depots)  is 
based  on  the  premise  that  it  is  reasonable  to  expect  the  support  process  to  maintain 
designed  levels  of  capability  through  the  equipment’s  service  life.  An  exception  to  this 
sometimes  occurs,  as  with  aircraft  engines  that  may  have  limits  placed  on  operational 
performance  to  prolong  periods  of  operational  service  between  maintenance.  In  the 
event  of  such  derating,  a  fault  would  be  defined  relative  to  the  derated  level  of  perfor¬ 
mance  rather  than  the  designed  level. 
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term  Type  B  faults.  A  Type  A  fault  is  one  where  symptoms  are  observ¬ 
able  no  matter  when  or  where  the  faulty  item  of  equipment  is  operated 
or  tested.  Faults  with  such  stationary  observability  have  the  kind  of 
visibility  that  is  needed  for  the  traditional  measures  of  R&M  to  be 
meaningful.  Special  data  collection  efforts  suggest  that  the  acquisition, 
product  improvement,  and  support  processes  have  come  a  long  way  in 
addressing  Type  A  faults.  Indeed,  much  of  the  current  emphasis  on 
improving  weapon  system  R&M  should  contribute  further  to  improving 
the  situation  with  Type  A  faults.  However,  equipments  frequently 
afflicted  with  Type  B  faults  also  need  to  be  addressed,  requiring  a 
whole  new  approach  to  how  we  view  R&M. 

The  new  approach  must  deal  with  both  Type  A  and  Type  B  faults 
and  it  must  provide  full  visibility  of  the  two  fundamental  phenomena 
that  determine  R&M: 

•  Fault  initiation, 

•  Fault  removal. 

To  do  this,  we  propose  a  view  of  R&M  based  upon  a  pair  of  considera¬ 
tions: 

•  The  frequency  with  which  new  faults  initiate  within  equipment, 
thereby  degrading  the  equipment’s  ability  to  dependably  deliver 
the  full  measure  of  its  designed  performance. 

•  The  efficiency  with  which  maintenance  technicians  remove 
faults,  thereby  restoring  the  equipment’s  full  measure  of 
designed  performance. 

To  apply  such  considerations  to  an  assessment  of  the  R&M  situation 
for  a  particular  subsystem,  the  Air  Force  must  be  able  to  detect  degra¬ 
dations  in  its  performance.  In  many  instances,  however,  the  Air  Force 
cannot  do  this  in  the  environment  of  routine  operations.  Instead  it 
must  rely  on  indications  gleaned  from  both  pilot  observations  and 
BIT-detected  symptoms.  Although  an  indication  of  difficulty  from  a 
single  flight  may  not  provide  satisfactory  evidence  of  a  fault,  it 
nonetheless  needs  to  be  documented  and  interpreted  in  the  context  of 
any  related  indications  for  important  patterns  to  be  recognized. 

When  a  pattern  develops,  it  must  be  stopped  as  early  as  possible 
even  though  that  may  require  extraordinary  actions6  by  the  support 
process.  To  minimize  the  cost  and  disruptions  created  by  special 
actions,  repetition  of  such  patterns  must  be  minimized  by  focusing  the 

6One  example  of  extraordinary  action  is  testing  a  unit  on  a  special  test  bench  (called 
a  system  bench  or  hot  mock-up).  Another  example  is  enclosing  a  unit  in  an  environmen¬ 
tal  chamber  during  testing. 
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product  improvement  process  on  rectifying  their  root  causes.  Likewise, 
to  minimize  the  cost  of  product  improvement  efforts,  the  need  for  such 
improvements  need  to  be  minimized  by  actions  during  the  acquisition 
process  that  cause  new  equipment  and  its  support  process  to  be 
designed  to  decrease  susceptibility  to  repeated  patterns  of  degraded 
performance. 


TRADITIONAL  AND  NEW  VIEWS  COMPARED 

The  value  of  information  added  by  the  new  view  is  illustrated  by  a 
set  of  six-month  case  histories  compiled  from  the  1984  data  collection 
phase  of  the  F-15/F-16  Radar  R&M  Improvement  Program.  The  case 
histories  illustrate  the  radar  R&M  situation  from  several  perspectives: 

•  For  a  radar  subsystem, 

—  Force-level  view  of  subsystem  experience, 

—  Aircraft-level  view  of  subsystem  experience  for  a  problem- 
plagued  aircraft, 

•  For  the  most  troublesome  type  of  LRU  in  a  radar  subsystem, 

—  Squadron-level  view  of  experience  with  that  type  of  LRU, 

—  Aircraft-level  view  of  experience  with  problem  plagued  LRUs 
of  that  type. 

Views  of  Radar  Subsystem  Experience 

Force-Level  View.  From  a  force-level  perspective,  Figs.  1  and  2 
provide  two  views  of  R&M  for  each  of  two  radars.  Figure  1  summa¬ 
rizes  radar  experience  for  150  aircraft  assigned  to  two  representative 
wings  of  F-15  C/D  aircraft.7  Figure  2  provides  a  similar  summary  for 
150  aircraft  assigned  to  two  representative  wings  of  F-16  A/B  aircraft.8 

The  top  bar  in  each  figure  portrays  a  traditional  view  based  upon 
contractor  calculated  estimates  for  MTBF.  The  estimates,  derived 
from  the  Air  Force’s  standard  MDC  system,  portray  only  part  of  the 
R&M  situation.  A  richer  view  of  the  situation  is  depicted  by  the  bot¬ 
tom  three  bars  in  each  figure. 

Although  the  mean  flight  hours  between  units  failing  shop  tests  is 
fairly  close  to  the  MTBF  estimates,  the  average  flying-hour  intervals 
between  pilot  requests  for  maintenance  were  much  shorter  than  those 
between  failure  of  shop  tests.  The  difference  is  caused  by  two  kinds  of 
problems:  shortcomings  in  fault  isolation  efficiency  and  ill-founded 

7The  1st  TFW  at  Langley  Air  Force  Base,  Virginia,  and  the  36th  TFW  at  Bitburg  Air 
Base,  FRG. 

“The  60th  TFW  at  Hahn  Air  Base,  FRG,  and  the  388th  TFW  at  Hill  AFB,  Utah. 


11 


F-15  C/D 
standard 
data 


Mean  flight  hours  between 
•  Unit  failed  shop  test 


•  Pilot  requested 
maintenance 


•  Flight  with  indicated 
fault 


Radar  R&M 
(  Improvement 
Program 
data 


J 


0  25  50  75  100 

Mean  flight  hours 


Fig.  1 — Standard  F-15  C/D  data  show  a  partial  picture  of  radar  R&M 


requests  for  maintenance.  The  F-15/F-16  Radar  R&M  Improvement 
Program  data  collection  and  engineering  analysis  demonstrated  that 
the  dominant  problem  is  fault  isolation  efficiency  rather  than  pilot’s 
demanding  maintenance  without  a  proper  basis.9  Indeed,  as  the  bottom 
bars  in  Figs.  1  and  2  show,  pilots  are  encountering  flights  with  indica¬ 
tions  of  faulty  radar  operation  at  average  flying-hour  intervals  much 
shorter  than  the  intervals  between  requests  for  maintenance.  Pilots 
may  be  too  tolerant  of  radar  subsystems  that  are  manifesting  signs  of 
performance  less  than  the  full  designed  capabilities  of  the  equipment. 
A  view  from  the  aircraft  level  sheds  some  light  on  this  possibility. 

Aircraft-Level  View.  Figure  3  summarizes  nearly  two  months  of 
radar  experience  for  an  F-16A  aircraft  (number  0577)  that  had  persist¬ 
ing  radar  problems  throughout  the  six-month  case  history  period.  The 
portrayed  experience  includes  pilot  observed  reports  from  the  BIT- 
detected  faults  (col.  2)  as  well  as  the  pilot’s  own  independent  observa¬ 
tions  and  assessment  of  radar  performance  (col.  3).  Column  4  (“radar 
op.”)  shows  the  pilot’s  net  assessment  of  radar  capability,  as  reported 
to  the  contractor  data  collectors,  and  col.  5  (“radar  code”)  shows  the 


9See  Gebman,  Shulman,  and  Batten,  1988  for  details. 
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Fig.  2 — Standard  F-16  A/B  data  show  a  partial  picture  of  radar  R&M 


equipment  status  code  that  the  pilot  reported  to  the  maintenance 
debriefer.  When  a  status  code  of  1  is  reported,  maintenance  assumes 
that  the  subsystem  does  not  need  maintenance.  A  code  2  or  a  code  3 
constitutes  a  pilot  request  for  maintenance,  with  code  3  indicating  a 
greater  degree  of  urgency  (fix  before  next  flight). 

This  figure  raises  three  major  points: 

1.  Pilots  do  not  request  maintenance  for  every  indication  of  a 

fault  in  the  radar.  A  request  for  maintenance  is  influenced  by 

many  considerations  including: 

•  What  does  the  airplane  need  to  do  on  its  next  training 
flight? 

•  Would  the  indicated  fault  affect  the  training  value  of  the 
next  fi:  gnt? 

•  What  are  the  chances  that  the  indicated  fault  was  caused 
by  microwave  signals  generated  by  something  external  to 
the  aircraft? 

•  What  is  the  likelihood  that  technicians  can  duplicate  the 
fault  on  the  ground? 

•  If  the  fault  is  duplicated,  what  are  the  chances  that  a 
replacement  spare  is  available? 
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2.  Standard  data  systems  miss  many  indications  of  degraded 
radar  performance  because  information  about  performance 
degradations  is  collected  only  when  a  pilot  requests  mainte¬ 
nance  (flights  in  which  returning  pilots  assign  a  Code  2  or 
Code  3  to  the  radar’s  performance).  However,  a  history  of 
degraded  performance  may  be  developing  even  when  pilots  are 
not  requesting  maintenance  (Code  1). 

3.  Although  the  F-16’s  BITs  are  very  reliable10  compared  with 
those  of  the  F-15,11  they  nevertheless  often  fail  to  detect 
faults,  especially  Type  B  faults.  Numbers  in  the  second 
column  of  Fig.  3  refer  to  the  specific  BITs  that  the  radar 
failed.  Some  faults — such  as  ones  causing  false  targets  and 
lock  problems — may  not  be  caught  by  BITs.  Others — such  as 
a  faulty  transmitter — are  only  sometimes  caught  by  BITs.  In 
Fig.  3,  for  example,  the  first  BIT  indication  of  a  fault  in  the 
transmitter  occurred  on  June  25,  when  tests  032  and  035 
failed.  Although  the  pilot  also  observed  indications  of  a  prob¬ 
lem  with  false  targets,  he  did  not  request  maintenance.  BIT 
indications  of  a  fault  in  the  transmitter  occurred  again  on  the 
next  day,  and  the  pilot  requested  maintenance.  However,  the 
radar  specialist  Could  Not  Duplicate  the  BIT  failures  reported 
by  the  pilot,  and  the  technician  did  not  remove  the  radar 
transmitter.  The  first  flight  the  following  day  yielded  no 
reports  of  any  BIT  failures,  but  the  next  flight  did,  and  the 
BIT  failures  again  pointed  to  a  faulty  transmitter.  Moreover, 
the  pilot  assigned  a  Code  3  to  the  radar.  Maintenance  then 
removed  the  transmitter  (XMTR)  and  sent  it  to  the  shop  for 
different  (and  more  detailed)  testing  and  repair.  The  replace¬ 
ment  transmitter  appears  to  have  operated  as  it  should  for  the 
remaining  five  months  because  the  BIT  indicated  no  subse¬ 
quent  faults  with  the  transmitter. 

Unfortunately,  F-16  number  0577  had  experiences  like  those  indi¬ 
cated  in  Fig.  3  during  the  entire  six-month  data  collection  effort.  This 
airplane  had  more  than  its  share  of  false  target  and  target-lock-on  dif¬ 
ficulties.  As  Fig.  3  illustrates,  there  were  many  flights  after  which  the 
pilot  reported  status  code  1  even  though  there  were  signs  of  faulty 

l0Tbi8  was  determined  by  tracking  the  movement  of  LRUs  from  airplane  to  shop  to 
airplane  and  monitoring  BIT-detected  faults.  When  the  shop  failed  to  fix  a  BIT- 
detectable  fault,  the  trail  of  faults  would  follow  the  LRU  to  the  next  airplane.  When  the 
contractors  subjected  faults  to  special  tests,  they  found  faults.  From  this  we  conclude 
that  when  the  F-16  radar’s  BIT  detects  a  fault,  it  is  a  reliable  indication  that  the  system 
has  a  fault. 

nGebman,  Shulman,  and  Batten,  1988,  Sec.  IV. 
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3 
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28 
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2 
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tgts  and  chevrons.  ACM  and 
siewable  trouble  locking 

Degrd 

1 

10 

False  tgts  and  chevrons. 

Radai  good 

OK 

1 

11 
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OK 

1 

13 

False  targets  /would  not  hold 
lock.  O  band  noise  bar  missing 

Degrd 

2 

(CNDI 

13 

Numerous  false  targets 

24 

Radar  slow  to  lock  on  —  had 
some  false  targets  and 
chevrons 

Degrd 

1 

29 

Rdr  w/n  lock  until  17-18  miles- 
m/then  break,  relock  and 
worked  OK 

OK 

1 

30 

Had  lots  of  false  targets 

Degrd 

1 

31 

Numerous  false  rgts 
throughout  flight 

Degrd 

1 

2 

Radar  cursors  were  stuck  on 

OK 

3 

2 

Numerous  false  targets 

Degrd 

1 

6 

001 

OK 

1 

Fig.  3 — Performance  of  radar  in  F-16  number  0577, 
June  20  through  August  6,  1984 


operation  and  even  though,  at  times,  the  pilot’s  net  assessment  was 
that  the  radar  subsystem  was  degraded  (col.  4).  This  degraded  condi¬ 
tion  remains  essentially  invisible  to  the  measures  used  to  support  the 
traditional  view  of  R&M  because  the  only  event  in  Fig.  3  that  the  tra¬ 
ditional  view  could  recognize  would  be  the  replacement  of  the 
transmitter  on  June  27.  The  new  view  strives  to  portray  the  full  situa¬ 
tion:  The  radar  was  struggling  to  deliver  its  designed  capabilities  for 
more  than  two  months  of  operations.  However,  these  problems  were 
rarely  brought  to  the  attention  of  maintenance.  Moreover,  the  radar 
contractor  would  never  have  known  about  these  problems  if  its  person¬ 
nel  had  not  been  present  to  debrief  the  pilots. 
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Because  these  problems  often  cannot  be  duplicated  on  the  ground, 
maintenance  technicians  will  often  assign  a  “CND”  to  the  problem  and 
wait  to  see  what  the  next  pilot  does.  The  data  show  that  the  next  pilot 
often  chooses  not  to  request  maintenance,  and  maintenance  personnel 
therefore  often  do  not  realize  that  the  radar  is  still  experiencing  prob¬ 
lems. 

The  pilot’s  net  assessments  in  Fig.  3  are  not  without  shortcomings, 
however.  A  major  limitation  is  that  the  pilot  cannot  keep  an  eye  on  all 
aspects  of  radar  operation  all  of  the  time.  The  ability  of  BIT  to  test 
and  monitor  the  performance  of  a  radar  subsystem  and  the  individual 
LRUs  in  that  subsystem  is  an  important  source  of  further  information 
about  the  status  of  equipment.  It  is  a  key  source  of  information  for  the 
new  view,  as  the  following  examples  illustrate. 

Views  of  Experience  with  a  Troublesome  Type  of  LRU 

The  LRU,  the  low  power  radio  frequency  (LPRF)  unit,  was  found  to 
be  the  dominant  R&M  problem  for  the  radar  on  the  F-16  A/B.12 

Squadron-Level  View.  Figure  4  illustrates  a  typical  squadron’s 
experience  with  this  type  of  LRU  in  the  28  aircraft  that  the  squadron 
operated  during  the  six-month  case  history  period.13  Each  aircraft’s 
flight  experience  is  represented  by  a  horizontal  bar  that  starts  (and 
ends)  on  the  date  the  aircraft  first  (and  last)  flew  during  the  case  his¬ 
tory  period.  For  12  of  the  aircraft  the  figure  portrays  a  completely 
clear  bar  signifying  that  the  BIT  detected  zero  faults  in  the  LPRF  dur¬ 
ing  the  six-month  case  history  period.  For  the  remaining  16  aircraft, 
one  or  more  segments  of  each  aircraft’s  bar  is  darkened  to  signify 
multiple-flight  episodes  during  which  the  BIT  was  reporting  to  the 
pilot  that  it  had  detected  a  fault  in  the  LPRF  LRU.  On  average, 
approximately  one-third  of  the  aircraft  at  any  given  time  were 
experiencing  an  episode  where  the  BIT  was  detecting  faults  in  the 
LPRF.  The  darkened  portions  of  the  bars  are  long  because  pilots 
waited  to  request  maintenance  and  maintenance  technicians  encoun¬ 
tered  so  much  difficulty,  first  to  verify  fault  existence  and  second  to 
determine  the  precise  location  of  faults.  These  problems  may  be  fully 
depicted  with  a  look  at  individual  aircraft’s  experience. 

Aircraft-Level  View.  A  particularly  rich  and  interesting  example 
is  provided  by  F-16A  aircraft  number  0752,  which  had  problems  caused 
by  LPRF  LRUs  throughout  the  six-month  period  and  beyond  until  the 

l2See  Sec.  IV  and  for  further  details  see  the  discussion  of  the  F-15/F-16  Radar  R&M 
Improvement  Program  in  Gebman,  Shulman,  and  Batten,  1988. 

l3The  10th  Tactical  Fighter  Squadron  in  the  388th  TFW. 
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Fig.  4— LPRF  BIT  failure  episodes:  10th  TFS, 
June  through  December  1984 


radar  contractor  helped  the  Air  Force  resolve  the  problem  during  a  spe¬ 
cial  follow-up  effort.  Evidence  of  the  LPRF-caused  problems  is 
detailed  in  Figs.  5  and  6. 

This  aircraft  encountered  two  serious  episodes  where  there  were 
clear  patterns  of  faulty  operation  with  its  radar  during  1984.  Figure  5 
shows  many  flights  where  the  BIT  detected  symptoms  of  the  faults;  it 
also  shows  many  flights  where  it  did  not.  This  illustrates  the  imperfect 
nature  of  BIT.  Unfortunately,  BIT  indications  have  many  omissions, 
and  faults  may  exist  in  a  unit  but  the  BIT  may  not  always  detect  their 
symptoms.  However,  when  the  BIT  in  the  F-16  A/B  radar  does  detect 
a  fault,  it  is  trustworthy,  even  if  it  fails  to  detect  a  symptom  on  the 
next  flight.14 

Arrowheads  in  the  top  row  of  Fig.  5  represent  flights  in  which  the 
BIT  detected  no  symptoms  of  faults  in  the  LPRF.  Arrowheads  in  the 
second  row  represent  flights  in  which  the  BIT  did  detect  symptoms. 
The  number  in  each  arrowhead  represents  a  set  of  consecutive  flights. 


uGebnaan,  Shulman,  and  Batten,  1988. 
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Fig.  5— History  of  LPRF  BIT  indications  of  degraded  performance: 
F-16  number  0752,  June  through  December  1984 


The  figure  shows  that  the  BIT  would  detect  symptoms  of  faults  during 
some  flights  but  not  during  others. 

Figure  5  shows  an  irregular  pattern:  Two  consecutive  flights  occur 
with  BIT-detected  symptoms,  one  without,  four  with,  one  without,  and 
so  on.  Here  we  have  an  illustration  of  one  or  more  faults  with  irregu¬ 
lar  symptoms,  typical  of  Type  B  faults.  Except  for  a  28-flight  respite 
during  August,  patterns  persisted  throughout  the  six-month  data  col¬ 
lection  period.  The  28-flight  respite  occurred  after  a  pilot  requested 
maintenance  and  radar  technicians  replaced  the  LPRF. 

The  replacement  happened  on  July  19,  1984,  after  the  fault  had 
deteriorated  to  the  point  where  a  circuit  board  in  the  LPRF  unit 
started  generating  smoke  that  entered  the  cockpit.  Smelling  the 
smoke,  the  pilot  aborted  the  flight  before  takeoff.  Unfortunately, 
analysis  of  these  and  other  data  obtained  after  the  radar  contractor 
and  the  Air  Force  had  finally  installed  another  LPRF  (during  January 
1985)  revealed  that  on  July  19,  1984  maintenance  technicians  probably 
had  replaced  the  smoking  LPRF  unit  with  another  that  was  also 
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defective.  Although  from  the  perspective  of  the  BIT  there  were  28 
flights  in  a  row  without  any  indications  of  a  fault,  Fig.  6  illustrates  the 
degraded  performance  observed  by  the  pilot  and  compares  them  with 
BIT  indications  on  the  same  flight.  A  BIT  indication  of  “no”  means 
the  BIT  found  no  fault,  while  a  “yes”  indicates  a  fault.  On  eight  occa¬ 
sions  when  the  BIT  indicated  no  fault,  the  pilot  observed  a  problem 
with  the  radar.  Even  on  most  of  the  occasions  when  the  pilot  observed 
a  problem,  the  flight  received  an  equipment  status  code  of  1  (mainte¬ 
nance  not  needed)  rather  than  a  2  (maintenance  needed  but  may  be 
postponed)  or  a  3  (maintenance  required  before  next  flight). 

Because  maintenance  technicians  were  being  apprised  of  neither 
BIT-detected  faults  from  code  1  flights  nor  pilot  comments  noted  in 
Fig.  6,  the  technicians  had  no  way  of  knowing  that  they  had  installed  a 
faulty  LRU  on  July  19  until  the  code  3  request  for  maintenance  over 
three  months  later  on  October  26. 

The  example  of  aircraft  0752  is  particularly  rich  because  it  also  illus¬ 
trates  how  depriving  technicians  of  BIT  information  from  code  1 
flights  can  hinder  their  maintenance  efforts  once  maintenance  finally 
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3 
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1 

no 
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Fig.  6— History  of  LPRF  pilot  indications  of  degraded  performance: 
F-16  number  0752,  June  through  December  1984 
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is  requested.16  For  example,  on  October  26,  even  though  the  BIT  failed 
to  detect  a  fault  during  the  flight  that  day,  the  pilot  gave  the  flight  a 
status  code  3  because  the  radar  was  flooded  with  false  targets.  Mainte¬ 
nance  technicians,  unaware  of  the  history  of  BIT-detected  faults  on 
flights  throughout  the  month,  placed  total  reliance  on  their  execution 
of  the  BIT  on  the  ground,  which  failed  to  find  any  faults.  Moreover, 
because  faults  in  any  one  of  several  different  LRUs,  as  well  as  external 
phenomena,  can  cause  excessive  false  targets,  the  technicians  decided 
to  take  no  further  action  and  instead  waited  to  see  if  the  pilots  asked 
for  maintenance  again. 

A  month  later,  on  November  26,  after  many  more  flights  with  BIT- 
detected  faults  (Fig.  6),  a  pilot  asked  for  maintenance  following  a  flight 
where  he  witnessed  degraded  performance  and  the  BIT  detected  faulty 
operation  as  well.  Maintenance  technicians  had  no  knowledge  of  the 
other  flights  where  the  BIT  detected  faults.  All  they  had  to  work  with 
was  the  pilot’s  report,  the  BIT  report  from  the  one  flight  on  November 
26,  and  the  results  of  their  own  execution  of  the  BIT,  which  found  no 
faults.  Because  the  latest  evidence  (their  execution  of  the  BIT) 
revealed  no  faults,  they  took  no  further  action  and  instead  chose  to 
wait  and  see  if  the  pilots  would  repeat  their  request  for  maintenance. 
Pilots  chose  not  to  request  maintenance,  even  though  the  pattern  of 
flights  with  and  without  BIT-detected  faults  continued  into  January. 

Although  Air  Force  maintenance  technicians  where  unaware  of  the 
patterns  depicted  in  Figs.  5  and  6,  the  radar  contractor’s  engineering 
personnel  had  developed  a  high  level  of  interest  in  such  patterns  that 
were  afflicting  about  one-third  of  the  aircraft  at  a  given  time.  Anxious 
to  show  the  Air  Force  that  the  patterns  could  be  terminated  simply  by 
installing  a  healthy  LPRF,  the  radar  contractor  persuaded  the  Air 
Force  to  direct  the  removal  of  the  faulty  LPRF  from  aircraft  0752  and 
a  few  other  aircraft  during  January  1985.  Subsequent  tracking  of  these 
aircraft  for  an  additional  30  days  validated  the  contractor’s  proposed 
remedy. 

The  example  of  aircraft  0752  illustrates  the  problems  of  dealing  with 
a  Type  B  fault  that  is  more  easily  found  in  the  dr  by  observation  than 
by  the  BIT  on  the  ground.  If  maintenance  had  seen  the  history  of 
unarguable  evidence  presented  in  Figs.  5  and  6,  the  LPRF  unit  would 
have  been  removed  back  in  October,  if  not  sooner.  Mdntenance  needs 
this  kind  of  composite  history  to  better  guide  its  actions.  In  Sec.  Ill  we 


18This  reveals  a  shortcoming  of  the  traditional  view  of  R&M  that  tends  to  weight  the 
last  observation  most  heavily,  if  not  exclusively.  The  idea  has  been  that  the  only  symp¬ 
toms  that  matter  are  those  from  the  last  flight,  or  the  last  test.  The  new  view  suggests 
that  all  of  the  evidence  be  accumulated  and  considered. 
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discuss  an  experimental  prototype  of  a  system  designed  to  collect  and 
provide  such  useful  information. 

Figures  5  and  6  also  demonstrate  that  these  fault  symptoms  are  not 
only  intermittent  from  flight  to  flight,  but  their  impact  varies  over  time 
as  well. 

With  the  measurements  operative  under  the  traditional  view  of 
R&M,  the  data  from  aircraft  0752  yield  a  99.9  percent  FMC  rate,  an 
MTBF  of  94  hours,  and  an  MTTR  of  2  hours  (Fig.  7). 16  This  FMC  is  a 
very  good  rate  for  the  reliability  of  a  fire  control  radar  and,  indeed,  is 
close  to  representative  of  the  F-16  radar.  From  the  vantage  point  of 
our  new  view  of  R&M,  the  aircraft  and  pilot  endured  seven  months  of 


Traditional  view 


New  view 


FMC 
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99.9% 

■ 

fault  prone  operation 

MTBF 

I 

Two  faults 

94  hours 

■ 

28  reports 

MTTR 

2  hours 

per  fault  removed 

Fig.  7 — Traditional  view  of  R&M  versus  the  new  view, 
F-16A  number  0752  radar 


16MTTR  is  more  correctly  understood  as  “mean  time  to  complete  a  repair”  than  sim¬ 
ply  “mean  time  to  repair.”  In  many  instances  “repairing”  is  only  a  matter  of  removing 
one  LRU  and  replacing  it  with  another.  Actual  repair  of  the  LRU  takes  place  at  the 
shop  or  depot. 
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fault-prone  operation  w^en  the  full  designed  capability  of  the  radar 
was  not  available,  all  because  of  of  two  faulty  units. 

Throughout  this  period  there  were  only  two  requests  for  mainte¬ 
nance  and  only  one  LRU  removed.  Even  during  the  one  period  of  28 
flights  without  BIT  indications  of  faults,  the  pilot  observed  faults  that 
the  BIT  finally  picked  up;  so  it  is  reasonable  to  assume  that  there  were 
one  or  more  faults  in  the  replacement  LPRF.  It  is  also  reasonable  to 
believe  that  during  the  other  89  flights  there  were  one  or  more  faults  in 
the  LPRF  that  could  inhibit  the  radar  from  delivering  its  full  designed 
capability.  Although  only  41  flights  yielded  BIT  indications  of  faults, 
twice  that  number  of  flights  were  apparently  flown  with  one  or  more 
faults  present  in  the  LPRF. 

General  Observations  About  the  New  View 

The  new  view  provides  a  framework  for  recognizing  and  addressing 
problems  that  heretofore  have  been  masked  by  a  view  that  is  sensitive 
to  only  part  of  the  R&M  landscape.  It  offers  the  potential  of  a  better 
basis  from  which  the  Air  Force  could  strengthen  its  ability  to  manage 
R&M. 


III.  THE  SUPPORT  PROCESS 


From  the  perspective  of  the  new  view,  R&M  is  determined  by  two 
fundamental  characteristics: 

•  Fault  initiation, 

•  Fault  removal. 

The  first  characteristic  is  established  completely  by  events  within  the 
acquisition  process  and  the  product  improvement  process.  The  second 
characteristic,  however,  is  also  influenced  by  the  effectiveness  of  the 
support  process.  Indeed,  a  primary  purpose  of  the  support  process  is 
the  removal  of  faults.  Thus,  an  assessment  of  R&M-related 
weaknesses  in  the  support  process  needs  to  start  with  an  examination 
of  how  effectively  that  process  is  performing  in  terms  of  removing 
faults. 

Strengthening  the  support  process  fault  removal  capability  offers  the 
Air  Force  the  most  immediate  opportunities  to  strengthen  its  overall 
ability  to  manage  weapon  system  R&M.  Moreover,  because  current 
R&M  related  deficiencies  in  the  support  process  result  largely  from 
lack  of  information,  better  information  will  highlight  support  deficien¬ 
cies  that  need  to  be  addressed  by  both  the  product  improvement  and 
the  acquisition  process. 


WEAKNESSES  IN  THE  WEAPON  SYSTEM 
SUPPORT  PROCESS 

The  weaknesses  in  the  support  process  obscure  the  Air  Force’s  abil¬ 
ity  to  see  difficult  problems  that  seriously  degrade  the  ability  of 
maintenance  technicians  to  remove  faults  from  important  equipment. 
Although  the  Air  Force  has  struggled  with  the  symptoms  of  these  prob¬ 
lems  for  a  long  time,  progress  at  identifying  the  root  causes  has  been 
hampered.  Weapon  system  program  managers  and  the  senior  leader¬ 
ship  of  the  Air  Force  have  lacked  a  methodology  for  providing  good 
visibility  of  fault  removal  effectiveness. 

Assessment  of  Fault  Removal  Effectiveness 

We  developed  and  applied  two  methods  for  assessing  how  effectively 
the  support  process  removes  faults  from  subsystems  that  make  up  an 
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aircraft  weapon  system.  The  Air  Force  could  use  each  method  to 
improve  its  ability  to  track  and  manage  its  capability  to  remove  faults. 

The  first  method  provides  a  rough  estimate  that  is  based  on  data 
routinely  collected  by  the  Air  Force.  The  second  method  provides  a 
more  complete  estimate,  but  it  requires  a  special  data  collection  effort. 
That  effort,  however,  can  provide  engineers  the  data  they  need  to  iden¬ 
tify  the  root  causes  of  fault  removal  difficulties. 

Assessing  Fault  Removal  Effectiveness  with  Currently  Avail¬ 
able  Data.  This  method  uses  information  in  the  Air  Force’s  MDC 
system  to  examine  the  worldwide  experience  of  an  aircraft  weapon  sys¬ 
tem.  For  our  analysis,  we  applied  the  methodology  to  an  example  data¬ 
base  for  calendar  year  1985  for  the  F-16  A/B  force.1  Based  on  our 
research  involving  other  weapon  systems,2  we  believe  these  results  are 
representative  of  the  experiences  of  a  broad  class  of  contemporary  tac¬ 
tical  aircraft. 


No-repair-action  jobs 

Fig.  8 — Flight  line  repair  action  jobs  on  the  F-16  A/B  weapon  system, 
per  100  flights,  USAF  MDC  system,  1985 


'Petruschell,  Smith,  and  Kirkwood,  1987. 

2See  Gebman,  Shulman,  and  Batten,  1988  for  the  F-15;  Mclver,  1974  for  the  F-lll 
series;  Nelson  1974  for  the  A-7D;  and  Robinson  1967  and  1972  for  the  F-4. 
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The  methodology  has  three  main  parts:  (1)  organization  of  the  data 
by  maintenance  jobs,  (2)  classification  of  maintenance  jobs  according 
to  whether  a  repair  was  accomplished,  and  (3)  counting  the  number  of 
jobs  in  each  of  two  classifications: 

•  Repair-action  jobs, 

•  No-repair-action  jobs, 

for  each  major  subsystem  constituting  a  weapon  system.  For  the  pur¬ 
poses  of  this  methodology,  we  define  a  maintenance  job  to  include  all 
of  the  flight  line  maintenance  actions  that  are  part  of  the  maintenance 
organization’s  response  to  a  discrepancy  that  is  reported  against  a 
specific  subsystem. 

Application  of  the  methodology  to  the  example  database  yields  the 
results  portrayed  in  Fig.  8.  The  vertical  axis  represents  the  number  of 
repair-action  jobs  per  100  flights  for  each  type  of  subsystem  in  the  F-16 
A/B  weapon  system.  The  horizontal  axis  represents  the  number  of 
no-repair-action  jobs  per  100  flights.  The  figure  presents  a  data  point 
for  each  of  the  11  subsystems  that  had  the  most  jobs  per  100  flights. 
Sixteen  subsystems  had  less  than  one '  repair  action  job  and  less  than 
one  no-repair-action  job  per  100  flights.  These  subsystems  account 
only  for  a  small  portion  of  the  total  maintenance  jobs. 

This  figure  identifies  the  subsystems  that  account  for  the  most 
maintenance  jobs  and  highlights  the  subsystems  accounting  for  most  of 
the  no-repair-action  jobs.  For  example,  the  fire  control  system  had,  on 
average,  five  maintenance  jobs  per  100  flights  in  which  there  actually 
was  a  repair.  There  were  also  about  five  requests  for  maintenance 
action  that  did  not  result  in  any  repair;  maintenance  would  find  noth¬ 
ing  to  repair  because  the  BIT  did  not  duplicate  the  fault  on  the  ground 
that  it  found  in  the  air,  or  the  technicians  could  not  stimulate  symp¬ 
toms  or  duplicate  the  problem  reported  by  the  pilot. 

The  traditional  view  of  R&M  has  been  to  discount  requests  for 
maintenance  that  resulted  in  no  repair  action  because,  it  was  reasoned, 
the  BIT  might  have  given  a  false  alarm  or  the  pilot  may  not  have  been 
really  sure  of  what  he  saw,  and  so  on.  The  new  view,  however,  pro¬ 
poses  that: 

•  All  requests  for  maintenance  are  to  be  presumed  valid. 

Special  tests  conducted  by  contractor  personnel  support  this  fault- 
existence  presumption.  An  even  stronger  proposal  is  that: 

•  All  indications  of  the  existence  of  a  fault  are  to  be  presumed 
valid  until  the  tracking  of  system  performance  over  time  proves 
otherwise. 


Application  of  either  presumption  to  the  data  in  Fig.  8  allows  one  to 
estimate  the  fault  removal  efficiencies  displayed  in  Fig.  9.  About  half 
the  time  when  there  was  a  fault  in  the  fire  control  system,  mainte¬ 
nance  technicians  were  able  to  find  something  to  repair  on  the  ground, 
suggesting  a  ft  alt  isolation  efficiency  of  about  0.5— that  is,  50  percent. 
In  contrast,  the  lighting  system  had  averaged  2.25  repair-action  jobs 
per  100  flights  and  only  0.45  no  repair-action  jobs  per  100  flights,  sug¬ 
gesting  a  fault  isolation  efficiency  of  83  percent.3  A  burnt-out  light  bulb 
does  not  present  the  maintenance  problems  of  fire  control,  ECM,  or 
weapon  delivery.  The  pilot  sees  the  bulb  is  out,  maintenance  confirms 
this  easily  on  the  ground,  and  it  replaces  the  bulb.  The  other  systems 
are  more  likely  to  have  problems  that  will  manifest  clear  symptoms  in 
the  air  to  the  pilots  and  the  BITs,  but  not  to  maintenance  personnel 
on  the  ground. 

A  system  with  many  maintenance  jobs  per  100  flights  and  a  low 
fault  isolation  efficiency  warrants  further  investigation.  To  illustrate 


No  repair-action  jobs 


Fig.  9 — Flight  line  fault  isolation  efficiency  on  the  F-16  A/B  weapon 
system,  per  100  flights,  USAF  MDC  system,  1985 


32.25/(2.25  +  0.45)  -  0.83. 
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this,  we  go  back  to  the  fire  control  system  and  examine  its  constituent 
subsystems  using  the  same  methodology  and  display  format.  The 
results  in  Fig.  10  show  that  the  fire  control  system’s  fault  isolation  effi¬ 
ciency  is  dominated  by  the  fault  isolation  efficiencies  for  the  radar  and 
the  inertial  navigation  system  (INS). 

Using  data  currently  available  to  the  Air  Force,  our  methodology  has 
identified  two  of  the  seven  subsystems  constituting  the  fire  control  sys¬ 
tem  as  the  dominant  sources  of  trouble  with  the  F-16  A/B  weapon 
system’s  fault  isolation  efficiency.  However,  the  data  have  not  identi¬ 
fied  the  causes  of  the  problems.  That  requires  a  special  data  collection 
effort. 

Assessing  Fault  Removal  Effectiveness  with  Special  Data. 

For  the  radar,  a  special  data  collection  effort  took  place  during  1984  as 
part  of  the  F-16  A/B  Radar  R&M  Improvement  Program.  Such  data 
can  be  used  to  estimate  the  fault  removal  efficiency  for  the  radar,  as 
well  as  to  determine  the  causes  of  fault  isolation  problems. 

Figure  11  shows  us  the  fault  isolation  efficiency  not  only  on  the 
flight  line,  but  also  at  the  shop  and  depot  levels  of  maintenance.  The 
first  bar  (the  flight  line)  corroborates  the  MDC  system  data  used  in  the 


No  repair  action  jobs 

Fig.  10 — Flight  line  fault  isolation  efficiency  in  the  F-16  A/B  fire 
control  system,  per  100  flights,  USAF  MDC  system,  1985 
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At  each  maintenance  level 


Overall 


Fig.  11— Overall  fault  isolation  efficiency  for  the  F-16  A/B  radar, 
16,000  flights,  special  data  collection,  1984-1985 


previous  figures:  Fault  isolation  efficiency  on  the  flight  line  is  about  50 
percent  for  the  radar.  However,  that  efficiency  in  the  shop  was  also 
imperfect,  though  better  than  on  the  flight  line.  When  an  LRU  from 
the  radar  was  sent  to  the  shop,  68  percent  of  the  time  the  shop  exe¬ 
cuted  a  repair  action.  When  the  shop  pulled  a  shop  replaceable  unit 
(SRU)  and  sent  it  to  the  depot,  the  depot  was  able  to  execute  a  repair 
action  about  80  percent  of  the  time. 

The  special  data  collection  and  associated  engineering  analysis  effort 
verified  that: 

•  Nearly  all  of  the  pilot  reported  discrepancies  were  valid  obser¬ 
vations,  even  though  flight  line  technicians  could  not  duplicate 
the  pilot  observed  symptoms  for  half  of  the  pilot  reports. 

•  Nearly  all  of  the  LRUs  sent  to  the  shop  were  faulty,  even 
though  the  shop  could  find  faults  in  only  68  percent  of  the 
LRUs. 
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At  first  glance,  the  track  of  improving  efficiency  in  Fig.  11  might 
look  good.  However,  the  product  of  the  efficiencies  at  each  mainte¬ 
nance  level  is  not  very  good.  The  flight  line  efficiency  times  the  shop 
efficiency  times  the  depot  efficiency  yields  an  overall  fault  isolation 
efficiency  of  27  percent.  That  is,  for  every  four  requests  for  mainte¬ 
nance,  the  support  process  takes  about  one  repair  action. 

We  also  need  to  examine  fault  removal  efficiency.  Figure  12  shows 
that  fault  removal  efficiency  is  a  function  of  both  maintenance  requests 
per  faulty  flight  and  fault  isolation  efficiency.  Although  the  fault  isola¬ 
tion  efficiency  is  still  one  in  four,  the  special  data  collection  effort 
found  that  the  pilots  were  asking  for  maintenance  at  the  rate  of  once 
every  five  flights  in  which  they  saw  an  indication  of  a  problem.  In 
other  words,  for  every  hundred  times  a  pilot  saw  a  difficulty  with  the 
radar  subsystem,  such  as  in  aircraft  number  0752  discussed  in  the  pre¬ 
vious  section  (see  Figs.  1  and  2),  he  would  request  maintenance  20 
times.  So  the  fault  removal  efficiency  for  the  entire  subsystem  is  the 
product  of  the  rate  of  these  maintenance  requests  and  the  overall  fault 
isolation  efficiency.  For  the  F-16  A/B  radar  subsystem,  this  efficiency 
is  5  percent. 
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Fig.  12— Fault  removal  efficiency  for  the  F-16  A/B  radar, 
special  data  collection,  1984-1985 
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Investigation  revealed  that  the  problem  with  the  F-16  A/B  radar  lies 
primarily  with  the  Low  Power  RF  unit,  one  of  the  main  boxes  in  the 
radar.  Over  half  of  the  units  (58  out  of  98  LPRFs)  going  back  into  the 
airplanes  within  15  flights  after  their  release  from  the  shop  were  again 
failing  the  BITs  in  the  air.  This  rate  is  about  one-tenth  the  reliability 
of  that  LPRF  unit,  so  it  is  highly  unlikely  that  very  many  of  the  faults 
are  new  ones.  About  half  the  time  the  shop  looks  at  an  LPRF  unit,  it 
cannot  find  anything  to  fix. 

Evidence  that  the  units  were  faulty  came  from  two  sources:  tracking 
the  faulty  performance  of  units  when  they  returned  to  aircraft  follow¬ 
ing  their  shop  visits,  and  a  special  test  for  30  units  that  continuously 
failed  BIT  while  in  various  aircraft,  only  to  pass  the  shop’s  tests. 

The  30  units  that  could  not  be  fixed  in  the  shop  were  put  through  a 
special  test  program.  The  LRUs  were  subjected  to  a  more  challenging 
thermal  environment  and  certain  tests  were  repeated  more  often  than 
would  normally  occur  in  the  shop.  These  special  procedures  discovered 
faults  in  20  of  the  30  units.  Unfortunately,  the  special  test  program 
did  not  have  the  capability  to  impose  a  vibration  environment.  We  do 
not  know,  therefore,  whether  vibration  may  have  elicited  symptoms  of 
faults  in  the  remaining  10  units. 

Low  fault  removal  efficiency  is  not  a  problem  peculiar  to  the  F-16 
A/B  radar.  A  parallel  collection  of  special  engineering  data  was  made 
on  the  F-15  C/D  radar  over  a  six-month  period  at  two  bases.  Here, 
too,  researchers  found  serious  deficiencies  in  the  shop’s  ability  to  fix 
units:  32  units  were  sent  to  the  shop  five  or  more  times;  seven  of  these 
units  were  sent  eight  or  more  times.4  Such  patterns  are  due  to  limita¬ 
tions  in  the  shop’s  ability  to  find  the  faults  that  are  causing  in-flight 
problems.  Symptoms  of  these  limitations  are  reflected  in  the  following 
statistics: 

•  In  37  percent  of  all  LRUs  sent  to  the  shop,  the  shop  found  no 
fault. 

•  In  20  percent  of  all  LRUs  sent  to  the  shop,  the  shop  detected 
faults  and  implemented  repair  actions,  but  subsequent  engineer¬ 
ing  analysis  revealed  that  all  of  the  repair  actions  were  unre¬ 
lated  to  the  faults  that  occurred  in  flight. 

In  the  latter  case,  the  shop  technicians  found  something  to  do,  perhaps 
an  adjustment  or  a  minor  repair;  but  whatever  it  was,  it  was  irrelevant 
to  the  fault  condition  detected  in  flight.  The  technicians,  lacking 

4Over  this  period,  these  units  would  be  expected  to  make  one  visit  to  the  shop.  Even 
allowing  for  any  kind  of  reasonable  random  distribution  (such  as  a  Poisson  distribution), 
five  or  more  visits  is  clearly  excessive. 
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necessary  engineering  data  to  know  otherwise,  had  no  idea  that  their 
efforts  were  irrelevant. 

Figure  13  illustrates  the  consequences  of  inefficient  fault  removal  in 
the  radars  of  the  F-15  C/D  and  F-16  A/B.  In  the  case  of  the  F-15  C/D 
radar,  faults  were  indicated  in  33  out  of  every  100  flights  during  the 
special  data  collection  effort;  of  these,  85  percent  experienced  faults 
that  had  been  indicated  in  previous  flights.  In  the  case  of  the  F-16 
A/B  radar,  the  situation  looks  better:  Only  22  flights  out  of  100  had 
some  indication  of  difficulty;  but  95  percent  of  those  with  fault  indica¬ 
tions  are  old  faults. 

Figure  13  analyzes  consequences  from  the  operator’s  point  of  view. 
From  the  R&M  engineering  management  point  of  view,  Fig.  14  carries 
the  same  basic  message  but  it  addresses  the  basic  characteristics  that 
determine  R&M  and  ultimately  drive  the  consequences  observed  by  the 
operators. 

Figure  14  illustrates  current  fault  removal  capabilities  in  the  context 
of  the  overall  R&M  situation  for  the  F-15  C/D  radar  and  the  F-16  A/B 
radar.  This  type  of  display  illustrates  how  the  R&M  of  such  subsys¬ 
tems  could  be  improved  and  implements  the  new  view  of  R&M. 
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Fig.  13— Consequence  of  inefficient  fault  removal  in  the  F-15  C/D 
radar  and  F-16  A/B  radar,  per  100  flights,  special  data 
collection,  1984-1985 
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Along  the  vertical  axis  we  show  the  mean  time  between  shop  con¬ 
firmed  failure  to  indicate  reliability.  To  complete  the  R&M  picture,  we 
propose  a  measure  of  fault  removal  efficiency  to  indicate  maintainabil¬ 
ity. 

As  Figure  14  shows,  the  F-16  A/B  radar  is  far  better  than  the  F-15 
C/D  radar  in  MTBF,  but  its  fault  removal  efficiency  is  worse.  In  both 
cases  the  fault  removal  efficiency  is  still  very  low,  confusing  the  differ¬ 
ence  between  reliability  and  maintainability.  This  composite  way  of 
viewing  R&M  is  therefore  intended  as  a  first-order  management  tool 
rather  than  a  contractual  device. 

When  a  system  or  subsystem  registers  a  low  percentage  for  fault 
removal  efficiency,  the  Air  Force  needs  to  examine  ways  to  improve  the 
fault  removal  capability  of  the  support  process.  The  Air  Force  may 
want  to  enlist  the  help  of  equipment  contractor  engineers  in  analyzing 
causes  of  fault  removal  problems.  For  instance,  they  may  find  that  the 
LPRF  unit  alone  is  causing  the  problem,  as  in  the  case  of  the  F-16  A/B 
radar,  or  that  the  BIT  in  several  LRUs  are  causing  problems,  as  in  the 
case  of  the  F-15  C/D  radar. 


“Shop  confirmed  Fault  removal  efficiency 

failures 

Fig.  14 — A  new  view  of  R&M  for  fighter  radars:  F-15  C/D  radar 
and  F-16  A/B  radar,  special  data  collection,  1984-1985 
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Support  Weaknesses  Contributing  to  Fault  Removal  Problems 

Beyond  equipment-specific  difficulties,  the  following  deficiencies  in 
the  support  process  are  strong  contributors  to  poor  fault  removal  per¬ 
formance.  The  current  process 

•  Provides  inadequate  information  about  avionics  equipment  per¬ 
formance  during  routine  training  flights.  Indeed,  pilot-observed 
indications  of  faults  often  go  unreported  to  technicians  even 
when  the  BIT  corroborates  degradation  in  the  equipment’s  per¬ 
formance. 

•  Fails  to  track  avionics  equipment  performance  by  serial  number. 
This  procedure  could  facilitate  the  identification  of  faulty  units 
that  repeatedly  circulate  between  shops  and  airplanes. 

•  Inadequately  integrates  maintenance  efforts  and  fault  informa¬ 
tion  across  maintenance  levels.  Even  though  avionics  techni¬ 
cians  use  different  tests  and  different  pass/fail  criteria  at  each 
maintenance  level  (flight  line,  shop,  and  depot),  technicians 
lack  test  dictionaries  that  would  enable  them  to  translate  test 
results  from  one  level  to  another. 

•  Provides  inadequate  capabilities  and  procedures  for  fixing  bad 
actor  equipment.  Not  only  do  technicians  lack  equipment  to 
represent  aspects  of  the  flight  environment,  but  they  also  lack 
capabilities  to  repeat  certain  tests  and  initiate  other  tests  before 
thermal  equilibrium  is  achieved.  Such  capabilities  would 
decrease  maintenance  time  and  lead  to  the  more  accurate  iden¬ 
tification  of  thermally  sensitive  faults. 


STRATEGY  FOR  IMPROVING  THE  SUPPORT  PROCESS 

To  strengthen  the  fault-removal  capability  of  the  support  process  we 
propose  a  four-part  strategy  that  includes  debriefing  pilots  for  indica¬ 
tions  of  faults,  tracking  avionics  subsystems  by  serial  number,  integrat¬ 
ing  repair  levels,  and  fixing  “bad  actors”  before  they  recirculate. 

Debrief  Pilots  for  All  Indications  of  Faults 

Pilots  need  to  share  with  maintenance  all  the  indications  of  difficul¬ 
ties  with  the  complex  weapons  system  equipment  that  they  perceive, 
large  or  small.  They  do  not  necessarily  have  to  request  maintenance 
every  time  they  notice  a  problem,  but  they  need  to  help  maintenance 
track  the  history  of  difficult  subsystems. 

To  improve  the  quality  of  information  received  at  postflight  debrief¬ 
ing,  the  Air  Force  could  develop  an  interactive  system  to  help  question 
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a  flight  crew  about  airplane  malfunctions.  Such  an  automated  system 
could  employ  data  transfer  units  to  capture  information  from  the  BIT5 
and  personal  computers  to  record  information  interactively  from  pilots. 
The  growing  complexity  of  avionics  equipment  and  avionics  faults 
makes  it  increasingly  difficult  to  obtain  a  thorough  characterization  of 
symptoms  and  operating  conditions  from  pilots  during  debrief. 

Track  Performance  of  Avionics  by  Equipment  Serial  Number 

The  tracking  of  problematic  subsystems  that  begins  with  pilot 
debriefs  should  continue  with  tracking  of  the  components  of  the  sub¬ 
systems  by  serial  number.  Maintenance  needs  to  keep  accurate  and 
updated  records  of  which  units  and  subunits  are  being  circulated 
between  the  flight  line  and  the  shop  and  depot.  This  step  is  essential 
in  helping  the  shop  and  the  depot  track  and  locate  units  they  are  hav¬ 
ing  difficulty  fixing.  The  tracking  of  units  helps  identify  the  bad  actors 
that  need  special  attention. 


Share  Information  About  Fault  Symptoms  Across 
Maintenance  Levels 

Because  the  support  process  must  use  different  tests  and  pass/fail 
criteria  at  each  maintenance  level,6  sharing  of  information  is  especially 
important  to  rectify  the  hard  to  fix  Type  B  faults.  One  beneficial  way 
to  improve  information  sharing  would  be  to  provide  test  translation  dic¬ 
tionaries  that  would  enable  avionics  technicians  at  one  maintenance 
level  to  translate  test  results  from  another  maintenance  level  into 
terms  they  would  find  useful  for  isolating  and  correcting  faults  and 
identifying  bad  actors.7  Technicians  must  use  information  from  the 
previous  level  to  confirm  that  they  are  correcting  failures  discovered  at 

5A  data  transfer  unit  is  a  data  recording  device  that  pilots  use  to  transfer  mission 
data  from  digital  computers  in  their  operations  facility  to  the  digital  computers  in  their 
aircraft  just  before  a  mission.  Many  modern  combat  aircraft  not  only  have  such  devices, 
but  they  also  use  these  devices  during  a  mission  to  record  faults  detected  in  flight.  When 
used  in  such  a  manner,  a  data  transfer  unit  is  a  painless  mechanism  for  obtaining  a 
thorough  and  accurate  record  of  BIT-detected  faults. 

^he  tests  on  the  aircraft  are  different  from  the  tests  run  in  the  shop,  which  in  turn 
are  different  from  tests  run  on  circuit  boards  at  the  depot.  Unfortunately,  these  differ¬ 
ences  are  necessary. 

7A  dictionary  could  relate  tests  from  one  level  to  another.  For  example,  each  BIT  at 
the  airplane  level  would  be  related  to  a  block  of  tests  at  the  next  higher  maintenance 
level  (the  shop).  For  example,  BIT  X  in  the  aircraft  is  related  to  the  block  of  Tests  A,  B, 
C,  and  D  in  the  LRU,  but  it  does  not  indicate  which  of  those  four  should  be  run  in  the 
shop.  Test  A  in  the  LRU,  for  instance,  is  likewise  related  to  the  block  of  Tests  1,  2,  3, 
and  4  in  the  SRU  but  does  not  indicate  which  one  of  those  four  should  be  run  at  the 
depot. 
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the  previous  level.  If  tests  at  different  levels  produce  different  results, 
the  unit  may  be  a  bad  actor  and  thus  require  special  surveillance  by 
pilots  and  flight  line  personnel.  And  if  the  unit’s  anomalous  behavior 
persists,  it  should  be  sent  to  the  depot  for  special  testing  and  analysis. 

Fix  Bad  Actor  Equipment 

The  test  translation  dictionary  is  only  an  aid  for  improving  fault- 
isolation  and  identifying  bad  actor  equipment.  The  next  key  element 
of  the  strategy  is  to  repair  the  problem  units  or  components  so  that 
they  do  not  circulate  between  the  support  process  and  aircraft  in 
degraded  condition.  To  improve  the  efficiency  of  repairing  bad  actors, 
we  recommend  improved  fault-isolation  capabilities  based  upon  those 
measures  that  contributed  to  the  F-15/F-16  Radar  R&M  Improvement 
Program.  These  include: 

•  Direct  entry  into  test  sequences  for  specific  sections  of  lengthy 
ground  avionics  tests  at  both  the  base  and  the  depot  so  techni¬ 
cians  could  avoid  having  to  run  tests  in  an  invariable  predeter¬ 
mined  sequence.  Technicians  would  have  improved  ability 
especially  to  find  Type  B  faults  that  are  sensitive  to  time- 
varying  thermal  conditions. 

•  Loop  testing  for  specific  tests  at  both  the  base  and  the  depot 
would  enable  technicians  to  run  the  same  test  repeatedly, 
thereby  improving  the  prospects  of  catching  Type  B  faults. 

•  Special  environmental  and  system  bench  capabilities  for  depots 
would  enable  test  equipment  to  better  replicate  operational 
modes  (such  as  air-to-ground  mode)  and  environmental  settings 
(such  as  extreme  temperatures)  that  especially  influence  Type 
B  faults. 


AN  EXPLORATORY  APPROACH  TO  IMPLEMENTING 
TWO  CRITICAL  PROPOSITIONS 

The  whole  strategy  for  strengthening  the  fault  removal  capability  of 
the  support  process  depends  upon  two  critical  propositions:  improved 
debriefing  of  pilots  and  serial  number  tracking  of  equipment.  Neither 
proposition  is  new.  The  challenge  is  to  devise  an  approach  to  imple¬ 
mentation  that  leads  to  a  bearable  burden.  Recent  developments  in 
computer  science  technology  may  provide  an  opportunity.  To  explore 
this  possibility  we  have  worked  with  Aeronautical  Systems  Division 
Strike  System  Program  Office  (SPO)  to  develop  a  prototype  to  support 
fuller  debriefing  and  serial  number  tracking  of  equipment  at  the  air 
base. 
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The  prototype  capability  was  designed  to  lighten  the  workload  of 
pilots  and  maintenance  technicians  alike  by  identifying  bad  actor 
equipment  and  moving  it  from  the  air  base  to  the  depot.  This  mainte¬ 
nance  aide  has  been  named  PORTER,  which  stands  for  Performance 
Oriented  Tracking  of  Equipment  Repair.  The  principal  aim  of 
PORTER  is  to  restore  aircraft  and  their  subsystems  to  their  full  design 
capability  as  quickly  as  possible.  Currently,  the  PORTER  prototype  is 
implemented  on  a  set  of  PCs. 

Rationale  for  a  PORTER  Capability 

Even  with  the  best  engineering  efforts — including  a  formal  process 
aimed  at  maturing  reliability  and  maintainability— some  avionics  faults 
will  still  evade  detection  and  correction  at  various  points  in  the  support 
process.  Consequently,  avionics  technicians  need  special  information 
to  track  down  such  faults,  correct  them,  and  document  deficiencies  in 
the  support  process  that  allowed  these  faults  to  escape  detection. 

Flaws  in  avionics  equipment  can  escape  detection  for  several  rea¬ 
sons.  The  narrow  confines  of  the  aircraft  and  the  limitations  of  the 
BITs  make  it  impossible  to  test  all  equipment  thoroughly  when  the  air¬ 
craft  is  in  the  air.  Similarly,  the  inability  to  replicate  airborne  stresses 
makes  it  impossible  to  conduct  a  fully  realistic  test  of  all  equipment 
when  the  aircraft  is  on  the  ground.  Moreover,  when  equipment  is 
removed  from  the  aircraft  and  taken  to  shops  and  depots  for  more 
exhaustive  testing,  both  the  nature  of  the  tests  and  their  pass/fail  cri¬ 
teria  change. 

Because  test  equipment  is  often  inadequate  to  assess  deficiencies  in 
complex  avionics  equipment,  omissions  occur  in  covering  certain  failure 
modes.  Consequently,  faulty  equipment  circulates  through  the  support 
system  until  either  the  fault  develops  into  a  more  serious  problem 
(such  as  an  open  circuit)  or  maintenance  personnel  take  extraordinary 
measures  to  isolate  the  problem.  Faulty  equipment  recirculates  all  too 
frequently  with  sophisticated  airborne  electronics  equipment. 

Items  of  equipment  that  circulate  in  this  fashion  are  commonly 
labeled  “bad  actors,”  and  they  deny  pilots  regular  and  dependable 
access  to  the  equipment’s  full  designed  capability.  Fixing  them  places 
great  demands  on  the  support  process,  consuming  time  and  resources 
spent  on  personnel,  spare  parts,  and  test  equipment.  The  air  base  shop 
repeatedly  tests  the  equipment;  the  depot  needs  to  use  special  environ¬ 
mental  tests  and  test  benches.  It  is  especially  critical  that  these 
extraordinary  efforts  focus  on  solving  only  the  most  serious  problems. 
The  efficient  identification  of  such  equipment  is  at  the  heart  of  the 
PORTER  concept. 
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Unfortunately,  the  Air  Force’s  standard  data  systems  do  not  gather 
sufficient  information  from  flight-to-flight  performance  to  enable 
maintenance  personnel  to  identify  and  diagnose  ineffectual  mainte¬ 
nance  and  evolving  problems  with  sophisticated  avionics.  PORTER 
strives  to  fill  this  gap. 

Once  a  particularly  hard-to-fix  fault  has  been  cured,  when  it 
occurred  in  a  different  unit,  mainetenance  could  go  directly  to  the  solu¬ 
tion.  After  several  successes,  it  becomes  a  “well-known  problem,” 
which  is  a  candidate  for  a  more  permanent  cure  through  the  product 
improvement  process.  Institutionalization  of  PORTER  would  create 
such  a  situation. 

Functional  Description  of  a  PORTER  Capability 

Figure  15  shows  how  maintenance  information  currently  flows,  and 
Fig.  16  shows  how  it  ideally  would  flow  after  instituting  a  PORTER 
capability.8 


FLIGHTLINE  MAINT.  SHOP 


^RespondsS  ^  /Test/repairsS 

Jo  requesl/*[  L! HU  LRU  ) 


E 


LRU 

History 

Sheet 

I 


LRU 

History 

File 


Fig.  15 — Current  maintenance  information  flow 


8See  Gebman,  Shulman,  and  Batten,  1988,  for  a  fuller  description  of  PORTER. 
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PR  1  and  PR  1  are  printed  reports 


Fig.  16 — Maintenance  information  flow  after  instituting  PORTER 


PORTER  starts  with  a  full  debrief  from  the  pilot  that  includes 
descriptions  of  all  indications  of  potential  faults— even  though  the  pilot 
may  not  ask  for  maintenance.9  All  such  indications  are  tracked  over 
time  along  with  descriptions  of  all  maintenance  actions.  This 
integrated  tracking  provides  the  basis  for  rapid  identification  of  bad 
actor  equipment.  It  also  provides  a  better  information  base  for 
engineering  R&M  improvements  to  both  the  airborne  and  the  ground- 
based  support  equipment.10 

However,  because  PORTER’S  primary  purpose  is  as  a  maintenance 
aid,  its  mechanization  must  be  tailored  and  synchronized  to  events  in 
the  maintenance  process.  An  operational  wing  has  assisted  not  only  in 


Eventually  it  would  be  best  to  have  a  computer  aided  interactive  debriefing  session 
where  questions  put  to  the  pilot  would  be  tailored  in  response  to  his  previous  answers. 
Initially,  though,  it  would  be  a  substantial  improvement  just  to  gather  all  pilot  observa¬ 
tions  concerning  potential  problems. 

10PORTER  can  contribute  to  improved  engineering,  first  by  providing  a  more  com¬ 
plete  view  of  difficult  maintenance  problems  and  second  by  providing  a  baseline  data  col¬ 
lection  system  that  can  be  expanded  upon  to  support  special  field  data  collection  efforts 
such  as  the  ones  that  were  part  of  the  F-15/F-16  Radar  R&M  Improvement  Programs. 


testing  and  demonstrating  PORTER,  but  also  in  defining  and  experi¬ 
menting  with  its  functional  specifications.11 

Less  than  one  year  after  concept  formulation,  an  austere  PORTER 
capability  was  in  a  demonstration  and  evaluation  phase  at  Bitburg. 
Based  upon  a  favorable  reception  of  even  the  austere  capability,  RAND 
and  the  Air  Force  are  working  on  integration  of  a  PORTER  capability 
into  the  Air  Force’s  Core  Automated  Maintenance  System  so  as  to 
eliminate  any  duplication  of  data  entry  tasks. 


SUMMARY 

Improving  the  flow  of  information  within  the  support  process  is  the 
first  essential  action  to  improve  fault  removal  and  thereby  strengthen 
the  Air  Force’s  capability  to  manage  the  R&M  of  already  fielded  equip¬ 
ment.  Such  improved  information  flow  needs  to  present  a  balanced 
view  of  R&M  that  highlights  problems  with  both  fault  initiation  and 
fault  removal.  Only  with  such  a  view  will  the  Air  Force  fully  identify 
the  major  deficiencies  in  the  capability  of  the  support  process  to 
remove  faults.  Such  identification  is  also  key  to  improving  the  product 
improvement  process  and  the  acquisition  process. 

uThis  interactive  development  of  PORTER’s  experimental  prototype  has  occurred  on 
PCs  at  the  36th  TFW  at  Bitburg  Air  Base.  The  Strike  SPO  at  the  Aeronautical  Systems 
Division  provided  on-site  support  through  a  contract  with  Support  Systems  Associates, 
Inc.  The  RAND  Corporation,  as  the  architect  of  PORTER,  helped  coordinate  the  tech¬ 
nical  development  and  demonstration  of  the  prototype. 


IV.  THE  PRODUCT  IMPROVEMENT  PROCESS 


Each  of  the  Air  Force’s  product  improvement  programs  has  one  or 
more  of  four  goals: 

1.  Improve  safety  of  flight, 

2.  Increase  functional  performance, 

3.  Lower  costs  of  operations  and  support, 

4.  Raise  readiness  and  sustainability. 

Achievement  of  each  of  these  goals  requires  attention  to 

•  Fault  initiation  rates, 

•  Capability  of  the  support  process  to  remove  faults  effectively. 

Because  such  attention  is  especially  critical  to  the  third  and  fourth 
goals,  we  use  these  goals  to  focus  our  assessment  of  R&M-related 
weaknesses  in  the  product  improvement  process. 


WEAKNESSES  IN  THE  WEAPON  SYSTEM 
PRODUCT  IMPROVEMENT  PROCESS 

To  determine  how  effective  the  current  product  improvement  pro¬ 
cess  is  at  applying  resources  to  reduce  fault  initiation  rates  and  to 
increase  the  capability  of  the  support  process  to  remove  faults  effi¬ 
ciently  we  need  to  know  something  about  where  the  dominant  prob¬ 
lems  lie,  their  extent,  and  their  influence  on  costs,  readiness,  and  sus¬ 
tainability.  Knowledge  of  such  considerations  is  at  best  incomplete. 
This  is  in  itsell  reason  foi  concern  about  the  effectiveness  of  the 
current  product  improvement  process. 

Lack  of  such  knowledge  is  one  of  two  major  weaknesses  addressed  in 
this  section.  The  second  is  a  lack  of  timely  implementation  for  the 
most  critically  needed  improvements. 

Inadequate  Information  About  Dominant  R&M  Problems 

Although  symptoms  of  the  dominant  problems  may  be  known,  the 
product  improvement  process  lacks  full  knowledge  of  the  connection 
between  the  root  causes  and  the  symptoms.  Moreover,  a  single  prob¬ 
lem  often  manifests  a  diverse  set  of  symptoms,  giving  the  impression 
that  there  are  a  lot  of  lesser  problems  and  contributing  to  fragmen¬ 
tation  of  efforts  to  improve  the  situation. 
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To  illustrate  these  points  we  compare  the  state  of  knowledge  about 
the  dominant  R&M  problems  with  two  subsystems  before  and  after 
special  field  data  collection  and  engineering  analysis  efforts.  Each  of 
these  special  efforts  was  far  more  intensive  than  any  other  field  data 
collection  effort  that  occurs  in  the  normal  execution  of  the  product 
improvement  process.  The  subsystems  that  were  the  objects  of  these 
special  efforts  are  the  F-15  C/D  radar  and  the  F-16  A/B  radar.  The 
data  collection  and  field  analysis  efforts  occurred  during  1984  and  1985 
as  part  of  the  F-15/F-16  Radar  R&M  Improvement  Programs.  In  the 
case  of  each  radar,  perceptions  of  the  dominant  R&M  difficulties  were 
changed  considerably  when  the  radar  contractors  presented  the  results 
of  their  special  efforts.1 

To  illustrate  this  point,  we  next  discuss  the  dominant  R&M  prob¬ 
lems  for  the  F-15  C/D  radar  and  the  F-16  A/B  radar.  What  are  they? 
What  was  known  about  them  before  the  special  efforts?  Why  had  the 
normal  conduct  of  the  product  improvement  process  not  revealed  the 
important  findings  of  the  special  efforts? 

Dominant  Radar  R&M  Problems  for  the  F-15  C/D  and  the 
F-16.  The  special  data  collection  and  engineering  analysis  efforts 
revealed  that  the  major  R&M  problems 

•  For  the  F-15  C/D  radar  lie  in  the  design  of  the  BIT, 

•  For  the  F-16  A/B  radar  He  in  the  design  of  the  Low  Power 
Radio  Frequency  (LPRF)  LRU. 

In  the  F-15  case,  the  radar  contractor  concluded  that  the  three  dom¬ 
inant  design  problems  are 

•  Errors  in  BIT  logic  that  cause  false  indications  of  problems  and 
give  maintenance  erroneous  reports  on  the  location  of  faults, 

•  Too  few  test  points  to  resolve  ambiguities  about  the  location  of 
faults, 

•  Inadequate  data  processing  capacity  to  accommodate  BIT 
software. 

In  the  F-16  case  the  radar  contractor  concluded  that  the  two  dom¬ 
inant  design  problems  are 

•  A  circuit  board  that  is  prone  to  development  of  flight  sensitive 
faults, 

•  An  internal  packaging  of  the  LRU  that  makes  it  difficult  to 
maintain  the  integrity  of  delicate  radio  frequency  connections. 


'Gebman,  Shulman,  and  Batten,  1988,  Sec.  IV. 
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The  subject  circuit  board  (an  SRU)  has  a  large  number  of  sites  where 
faults  are  developing  because  of  molecular  migration.  At  these  sites, 
gold  connector  ribbons  are  joined  with  an  indium  solder.  Unfortu¬ 
nately,  a  chemical  process  is  taking  place  whereby  molecules  in  the 
connectors  are  being  drawn  steadily  into  the  solder.  Eventually, 
enough  molecules  migrate  in  this  fashion  to  degrade  the  quality  of  the 
electrical  connection.  Over  time,  the  quality  of  the  connection  will 
degrade  and  eventually  the  connector  ribbon  will  separate  from  the 
solder.  Such  an  open  circuit  is  far  easier  for  shop  and  depot  tests  to 
detect  than  is  an  intact  yet  degraded  connection. 

Inside  the  LRU,  many  delicate  radio  frequency  connections  are 
located  in  hard  to  service  areas  and  require  precise  alignment  and 
torque  to  function  properly.  These  connections,  many  of  which  must 
be  opened  when  the  LRU  receives  maintenance,  may  manifest  no  prob¬ 
lems  on  the  ground  during  shop  tests.  However,  if  the  connection  is 
not  within  tolerances,  it  may  encounter  difficulties  when  subjected  to 
dynamic  flight  loads. 

Knowledge  Before  Special  Efforts.  The  Air  Force’s  mainte¬ 
nance  data  collection  system  was  the  main  source  of  information  for 
the  product  improvement  process.  It  revealed  virtually  no  direct  evi¬ 
dence  of  the  BIT  problems  with  the  F-15  radar.  In  the  case  of  the 
F-16  radar,  this  source  led  to  the  assessment  of  symptoms  summarized 
in  Fig.  17,  which  summarizes  the  state  of  knowledge  and  the 
contractor’s  proposed  improvement  plan  before  the  special  efforts  in 
1984-1985.  The  story  summarized  in  Fig.  17  is  that  data  in  the  MDC 
system  located  a  problem  with  either  the  LPRF  unit  or  the  support 
process  for  that  unit.  According  to  the  MDC  system  data,  the  flight 
line  technicians  were  sending  many  LPRF  units  to  the  shop  that  shop 
technicians  could  find  no  faults  in. 

Before  the  special  data  collection  and  analysis  effort,  the  root  causes 
of  the  problems  with  the  LPRF  (or  its  support  process)  were  masked 
by  several  factors.  For  one  thing,  a  radar  with  a  99-hour  MTBF  does 
not  draw  much  attention,2  especially  when  it  is  not  known  that  there 
are  only  six  flying  hours  between  indicated  faults  (80  percent  of  which 
were  by  the  BIT).  Moreover,  false  targets,  target  tracking  problems, 
and  the  like  can  often  result  from  factors  external  to  the  aircraft. 
Furthermore,  the  extent  of  flight-sensitive  failure  modes  in  the  LPRF 
was  not  generally  known.  Many  of  the  faults  detected  by  the  BIT  were 
thought  finally  to  result  from  a  grounding  problem  that  could  be 
corrected  by  a  new  shim  that  would  provide  a  tighter  electrical  path  for 
grounding  the  radar  equipment  to  the  airframe. 


2The  F-4’s  radar  has  less  than  a  9-hour  MTBF. 
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LPRF  Antenna 
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Fig.  17— Knowledge  of  R&M  needs  for  the  F-16  A/B  radar 


This  hypothesis  was  rigorously  tested  during  1984  as  part  of  the 
F-16  Radar  R&M  Improvement  Program.  Special  tests  in  which  this 
new  shim  was  installed  showed  that  it  would  not  correct  problems  the 
BIT  was  attributing  to  the  LPRF:  Of  20  aircraft  that  received  new 
shims,  19  were  still  found  to  have  faulty  LPRF  units  that  caused  the 
BIT  fault  indications.3 

In  part,  problems  with  the  LPRF  resulted  from  squeezing  a  capable 
and  versatile  radar  into  a  very  small  volume  with  a  tight  weight  limitation 
and  a  limited  amount  of  environmental  cooling.  Advances  in  technology 
have  now  created  an  opportunity  to  develop  a  more  maintainable  LPRF 


3The  units  were  determined  to  be  faulty  as  a  consequence  of  two  considerations. 
First,  the  units  left  a  trail  of  BIT-detected  faults  that  followed  the  subject  units  from  air¬ 
plane  to  airplane.  Second,  although  the  units  had  histories  of  the  shops  failing  to  find 
any  faults,  the  contractor  engineers  and  technicians  participating  in  the  special  program 
were  able  to  subject  the  units  to  special  tests  that  revealed  faults  not  being  detected  by 
the  shop’s  normal  testing  procedures. 
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that  should  be  less  prone  to  the  kinds  of  connection  problems  that  plague 
the  current  design.4 

Knowledge  Limitations  of  the  Current  Product  Improvement 
Process.  The  dominant  R&M  problems  with  the  radars  in  the  F-15 
C/D  and  the  F-16  A/B  were  identified  and  documented  through  the 
special  data  collection  effort  in  the  F-16  A/B  Radar  R&M  Improve¬ 
ment  Program.  The  extent  and  severity  of  the  R&M  problems  with 
the  F-15’s  BIT  and  the  F-16’s  LPRF  were  determined  and  validated  by 
a  four-step  process  that  started  with  a  special  data  collection,  that  then 
guided  engineering  tests,  engineering  analysis,  and  independent  review. 

The  current  product  improvement  process  lacks  such  special  and 
intensive  data  collection  efforts.  Figure  3  helps  show  why  that  is  a 
serious  weakness. 

The  lack  of  a  continuing  record  on  how  individual  radars  perform 
creates  two  major  problems: 

•  Maintenance  resources  cannot  be  applied  to  the  radars  with  the 
greatest  need. 

•  Engineering  resources  cannot  be  applied  to  those  radar  R&M 
problems5  that  cause  the  greatest  potential  degradation  in  per¬ 
formance. 

As  with  the  radar  and  other  modern  avionics  on  contemporary 
fighter  aircraft,  the  LPRF  unit  rarely  experiences  a  complete  failure 
that  renders  it  totally  incapable  of  supporting  the  airplane’s  mission. 
Its  most  common  failure  modes  have  two  characteristics: 

•  They  degrade  the  available  level  of  performance  by  reducing 
detection  ranges,  introducing  target  tracking  problems,  and/or 
creating  false  targets. 

•  They  manifest  themselves  mainly  when  the  radar  is  exposed  to 
thermal  and  dynamic  stresses  from  the  airplane’s  operational 
environment.  Thus  the  BIT  will  frequently  catch  a  fault  during 
flight  but  later  fail  to  duplicate  it  when  the  airplane  is  on  the 
ground. 


4To  explore  this  idea,  the  Radar  R&M  Improvement  Program  commissioned  Westing- 
house  to  develop  a  conceptual  design  for  a  new  LPRF  drawing  on  technologies  that 
Westinghouse  is  currently  incorporating  in  its  contemporary  radars.  This  new  design 
enables  more  circuitry  to  be  packaged  into  a  smaller  volume.  Such  packaging  is  more 
modular  and  facilitates  disassembly.  With  connections  on  the  back  of  each  module,  shop 
technicians  would  no  longer  have  to  disrupt  so  many  delicate  radio  frequency  circuits  to 
remove  a  module. 

,vThe  root  causes  of  which  may  lie  in  the  radar  or  its  support  process. 
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To  further  illustrate  the  need  for  a  special  data  collection  and 
engineering  analysis  to  sort  out  dominant  factors,  we  now  turn  briefly 
to  the  second  example  in  Fig.  17.  Here,  the  Air  Force’s  MDC  system 
data  reported  that  the  antenna’s  harness  was  wearing  out.  It  simply 
was  not  lasting  very  long,  and  when  it  wore  out  maintenance  had  to 
send  the  antennas  to  the  depot.  During  1983  the  Air  Force’s  plan 
called  for  improving  the  reliability  of  the  harness. 

The  additional  engineering  data  and  the  general  review  of  the 
radar’s  R&M  situation  that  was  part  of  the  F-16  A/B  Radar  R&M 
Improvement  Program  led  to  a  different  perspective.  The  dominant 
R&M  problem  was  not  reliability,  it  was  maintainability.  The  harness, 
it  was  discovered,  could  be  redesigned  for  replacement  in  the  field  so 
that,  when  it  does  wear  out,  maintenance  would  not  have  to  send  the 
whole  LRU  to  the  depot.  Moreover,  it  turned  out  that  such  a 
redesigned  and  reconstructed  harness  would  also  wear  out  less  often. 
However,  the  rapidly  scanning  antenna  will  always  cause  severe  wear. 

The  F-15’s  BIT,  the  F-16’s  antenna,  and  its  LPRF  all  are  examples 
that  illustrate  a  general  lack  of  knowledge  in  the  product  improvement 
process  concerning  the  root  causes  for  dominant  R&M  problems. 
Knowledge  is  lacking  in  the  normal  process  because  equipment  con¬ 
tractors  are  not  involved  in  special  data  collection  efforts;  and  lacking 
contractor  supplied  engineering  analyses,  the  Air  Force  does  not  have  a 
sound  position  from  which  it  can  view  a  weapon  system’s  R&M  posture 
in  the  context  of  underlying  causes  of  problems.6 

Lack  Timely  Implementation  of  Most 
Important  Improvements 

Timely  implementation  of  the  most  important  R&M  improvements 
is  hindered  by  lacks  in  four  areas: 

•  No  mechanism  for  initiating  special  data  collection, 

•  No  consensus  about  root  causes  of  dominant  problems, 

•  Lack  of  continuity  in  weapon  system  program  management 
responsibility, 

•  Poor  coordination  of  management  of  weapon  system  after 
PMRT. 

Lack  of  a  Mechanism  for  Initiating  a  Special  Data  Collection. 
When  a  large  and  complex  system  that  is  built  from  many  leading  edge 

6A  PORTER  capability  would  help  improve  the  Air  Force’s  position  and  it  would  help 
contractors  carry  out  special  data  collection  efforts.  It  would  not,  however,  provide  an 
adequate  alternative  to  the  kind  of  contractor  involvement  needed  for  engineering 
analysis. 


technologies  reaches  its  initial  operational  capability  (IOC)  date,  it  is 
reasonable  to  expect  shortfalls  in  R&M  characteristics.  This  is  when 
the  process  of  maturing  and  improving  the  product  can  begin  in  ear¬ 
nest.  At  present,  however,  the  Air  Force  lacks  a  mechanism  for  initiat¬ 
ing  the  kind  of  special  data  collection  and  corollary  engineering 
analysis  that  is  essential  to  an  effective  maturation  program. 

The  need  for  a  formal  mechanism  is  illustrated  by  the  tedious  and 
time-consuming  process  that  was  required  to  establish  the  special 
efforts  for  the  F-15  and  F-16  radars. 

Figure  18  illustrates  the  current  timetable  for  R&M  maturation  of 
the  F-16  A/B  radar.  The  F-16  A/B  Radar  Maturational  Development 
Demonstration,  which  yielded  the  information  for  a  F-16  A/B  radar 
improvement  package  (presented  below),  was  first  discussed  in  Sep¬ 
tember  1980,  formally  proposed  at  the  beginning  of  1981,  and  approved 
six  months  later.  From  the  date  of  approval,  it  took  three  years  for  the 
Program  Management  Directive  to  come  through  and  for  all  the  con¬ 
tracting  to  be  completed.  Theoretically,  the  Data  Collection  and 
Analysis  phase  of  the  demonstration  began  in  1981  when  the  proposal 
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Fig.  18 — Pace  of  R&M  maturation  for  the  F-16  A/B  radar 
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was  approved;  but  in  fact,  execution  of  that  phase  did  not  begin  till 
1984  when  the  contract  was  let.  It  was  midway  through  1984  before 
data  collection  and  engineering  analysis  began  (see  cross-hatched  area 
on  “Execution”  phase).  When  that  stage  was  completed  in  mid- 1985, 
the  recommendations  in  the  improvement  package  were  fully  assem¬ 
bled.  At  that  point,  the  engineers  knew  what  needed  to  be  done  to 
mature  R&M  for  F-16  A/B  radar  and  its  support  process. 

If  such  a  special  effort  had  been  launched  three  to  four  years  sooner 
(shortly  after  IOC),  we  believe  that  the  major  results  would  have  been 
the  same,  because  the  problems  with  both  the  LPRF  and  the  antenna 
were  known  at  that  time.  What  was  lacking  at  that  point  was 
knowledge  of  the  extent  of  these  problems  in  relation  to  other  prob¬ 
lems.  A  thorough  data  collection  and  engineering  analysis  and 
independent  review  at  that  time  would  have  shed  much  light  on  the 
identity  of  the  dominant  R&M  problems  and  their  root  causes. 

Lack  of  Consensus  About  Root  Causes  of  Dominant  Problems. 
Essential  to  timely  implementation  of  improvements  is  a  consensus 
about  three  things: 

•  The  relative  importance  of  the  problem, 

•  Its  cause(s), 

•  The  most  cost-beneficial  remedy. 

To  build  a  consensus  on  such  matters,  the  Air  Force  needs  four  ele¬ 
ments:  (1)  contractor  involvement  in  engineering  analysis,  (2)  systems 
analysis  of  alternative  courses  of  action,  (3)  independent  review  to 
assure  objectivity,  and  (4)  active  participation  of  the  organizations  that 
must  contribute  to  the  consensus  and  accomplish  implementation.  The 
results  of  the  first  three  elements  for  the  F-15/F-16  Radar  R&M 
Improvement  Program  reveal  some  important  things  about  the  neces¬ 
sary  scope  and  conduct  of  such  special  efforts.  To  lay  the  foundation 
for  these  points  we  review  the  results  for  the  F-16  A/B  radar  and  then 
we  turn  to  the  F-15  C/D  radar. 

Redesign  of  the  LPRF  would  serve  as  the  cornerstone  in  implement¬ 
ing  an  R&M  maturation  program  for  F-16’s  radar.  A  complete  package 
of  R&M  improvements  for  the  F-16’s  radar  would  include  ..he  items 
listed  in  Table  1.  The  improvements  fall  into  three  priority  classifica¬ 
tions  based  on  subjective  assessments  of  their  potential  benefits.7 
Priority  2  improvements  for  the  F-16  consist  of 


7In  spite  of  these  classifications,  we  recommended  that  the  Air  Force  pursue  all  the 
identified  improvements. 
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Table  1 

A  MATURATIONAL  DEVELOPMENT  IMPLEMENTATION  PACKAGE 
FOR  THE  F-16's  RADAR 


Priority  1 

Priority  2 

Priority  3 

F-lo 

Specific 

LPRF 

Antenna  cable 

Radio  frequency 
test  station 

Depot  environmental 
testing  for  LPRF 

Technical  orders 
Shop  software 

Generic 

Tracking  equipment 
by  serial  number 
Test  translation 
dictionaries 

Direct  entry 
testing 

Loop  testing 

Material  deficiency 
reporting 
Technical  Orders 
feedback 
Interactive  pilot 
debrief 

1.  Redesigning  and  re-routing  the  antenna  cable  to  allow  replace¬ 
ment  at  the  AIS  and  thus  avoid  having  to  send  the  entire 
antenna  to  the  depot  for  cable  replacement 

2.  Using  recently  developed  technologies  to  improve  the  main¬ 
tainability  of  the  radio  frequency  station,  currently  the  most 
difficult  station  to  maintain  in  the  AIS 

3.  Instituting  environmental  testing  at  the  depot  to  improve 
repair  of  the  current  LPRF  and  other  bad-actor  equipment. 

Priority  3  improvements  for  the  F-16  consist  of 

1.  Changing  Technical  Orders  to  make  them  more  consistent, 
compact,  accurate,  and  useful  for  maintenance  personnel 

2.  Correcting  deficiencies  in  shop  software  that  allow  certain 
faults  to  escape  detection. 

Such  a  total  development  package  could  triple  the  mean  time 
between  indicated  faults  (from  six  hours  to  19  hours)  and  also  reduce 
the  maintenance  workload  by  35  percent.  The  total  development  pack¬ 
age  (excluding  Priority  3  items)  would  cost  $250  million,  although  some 
of  this  cost  would  be  offset  by  the  decreased  need  for  maintenance. 

Most  of  the  F-16  money  must  go  to  retrofitting  the  LPRF  unit.  The 
principal  benefit  to  be  derived  from  the  complete  package  is  a  substan¬ 
tial  increase  in  fault  removal  efficiency  from  about  7  percent  to  nearly 
55  percent  (see  Fig.  19).  The  dimension  of  MTBF  offers  less  benefit. 
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Fig.  19 — Opportunities  for  benefiting  from  R&M  maturation 


The  large  increase  in  fault  removal  efficiency,  however,  would  yield  a 
subtantial  increase  in  the  number  of  fault-free  flights.  Although  the 
precise  magnitude  of  the  increase  is  unclear,  because  we  don’t  know 
how  many  flights  were  flown  with  faults  that  generated  no  symptoms, 
we  can  estimate  the  potential  magnitude  by  assuming  a  ratio  for  the 
total  number  of  faulty  flights  to  the  number  of  flights  with  symptoms. 
Figures  5  and  6  present  an  example  where  there  were  two  fault 
episodes  that  affected  at  least  89  flights  (48  +  41),  although  the  BIT 
detected  faults  on  only  41  of  the  flights.  This  experience  suggests  that 
the  number  of  fault-plagued  flights  may  be  as  much  as  double  the 
number  with  faults  indicated  by  the  BIT.  On  that  assumption,  the 
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incorporation  of  the  improvement  package  would  increase  fault-free 
flights  from  56  to  86  percent.8 

The  dominant  problem  with  the  F-15  C/D  radar  is  the  BIT  itself, 
which  one-third  of  the  time  incorrectly  indicates  a  fault  when  there  is 
none.  Moreover,  even  when  there  is  a  real  fault  present,  one-third  of 
the  time  it  indicates  the  wrong  LRU  as  the  location  for  the  fault. 

To  deal  with  the  F-15’s  BIT  problems,  the  special  data  collection 
and  analysis  effort  of  the  Radar  R&M  Improvement  Program  identified 
three  kinds  of  needed  improvements: 

1.  Improvements  to  the  BIT  Software.  Although  new  tactical 
software  is  developed  and  tested  each  year,  the  BIT — which  is 
interleaved  with  this  tactical  software— is  not  retested.  Dur¬ 
ing  the  Radar  R&M  Improvement  Program,  Hughes  engineers 
found  ten  situations  in  which  the  BIT,  because  it  is  inter¬ 
leaved  with  the  tactical  software,  falsely  detects  faults. 

2.  Additions  to  the  BIT  Hardware.  During  the  data  collection 
and  engineering  analysis  stage,  the  Warner-Robins  ALC  spon¬ 
sored  a  program  to  develop  special  ground  support  equipment 
that  could  augment  the  BIT  function.  This  equipment  neces¬ 
sarily  requires  up  to  three  hours  for  use,  so  it  is  not  designed 
to  support  the  quick  turnarounds  required  in  wartime.  Even 
so,  it  could  still  be  used  during  the  night  when  the  demands 
for  quick  turnarounds  are  lessened.  Moreover,  it  could  be 
available  in  about  two  years. 

3.  Improvements  to  the  BIT  Hardware.  Adding  memory  and  test 
points  to  the  BIT  hardware  will  provide  more  of  the  capabili¬ 
ties  that  are  being  incorporated  in  the  ground  support  equip¬ 
ment  and  will  allow  quick  turns  without  further  ground  sup¬ 
port  equipment. 

Because  the  F-15  radar  first  entered  operational  service  in  1975,  it  is 
reasonable  to  enquire  why  the  BIT  deficiencies  had  not  been  resolved 
much  sooner.  A  major  factor  is  that  the  Radar  R&M  Improvement 
Program  provided  the  radar  contractor  the  first  opportunity  to  make 
an  in-depth,  first-hand  engineering  examination  of  what  was  happening 
in  the  field.  Although  the  radar  contractor  had  had  technical  represen¬ 
tatives  assigned  to  various  bases  over  the  years,  these  representatives 
could  not  begin  to  collect  the  kinds  of  data  needed  to  define  the 
specific  engineering  deficiencies  with  the  current  BIT. 

8An  alternative  assumption  of  1.5  faulty  flights  for  each  flight  with  a  fault  indicated 
means  that  the  incorporation  of  the  improvement  package  would  increase  fault-free 
flights  from  67  to  90  percent. 
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Improvements  to  the  BIT’s  software  and  hardware  would  be  the 
cornerstone  in  implementing  maturational  development  improvements 
for  the  F-15’s  radar.  A  complete  package  of  maturational  development 
improvements  for  the  F-15’s  radar  would  include  the  items  listed  in 
Table  2.  The  improvements  fall  into  three  priority  classifications 
based  on  subjective  assessments  of  their  potential  benefits.9  Priority  2 
improvements  for  the  F-15  consist  of 

1.  Undertaking  LRU  fixes  for  the  radar’s  exciter,  transmitter, 
receiver,  antenna,  analog  processor,  programmable  signal  pro¬ 
cessor,  and  power  supply 

2.  Adding  dynamic  test  and  fault-isolation  capabilities  for  the 
Antenna  A  Test  Station  in  the  AIS 

3.  Instituting  environmental  testing  and  adding  a  complete  radar 
test  bench  to  the  depot  equipment10 

4.  Undertaking  research,  evaluation,  and  selection  of  a  replace¬ 
ment  for  Coolanol,  which  is  prone  to  contamination. 


Table  2 

A  MATURATIONAL  DEVELOPMENT  IMPLEMENTATION  PACKAGE 
FOR  THE  F-15’8  RADAR 


Priority  1 

Priority  2 

Priority  3 

F-15 

Specific 

BIT  software 
and  hardware 

LRU  fixes 

Antenna  A  Test 
Station 

Depot  equipment 
Replacement  of 
Coolanol 

Technical  orders 
Shop  software 

Reseat  problems 

Generic 

Tracking  equipment 
by  serial  number 
Test  translation 
dictionaries 

Direct  entry 
testing 

Loop  testing 

Material  deficiency 
reporting 

Technical  Orders 
feedback 
Interactive  pilot 
debrief 

9In  spite  of  these  classifications,  we  recommend  that  the  Air  Force  pursue  all  the 
identified  improvements. 

10The  test  bench  allows  testing  of  any  radar  LRU  as  part  of  the  overall  radar  subsys¬ 
tem. 
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Priority  3  improvements  for  the  F-15  consist  of 

1.  Changing  Technical  Orders  to  make  them  more  consistent, 
compact,  accurate,  and  useful  for  maintenance  personnel 

2.  Correcting  deficiencies  in  shop  software  that  allow  certain 
faults  to  escape  detection 

3.  Identifying  reasons  why  problems  with  SRU  connections  seem 
to  disappear  when  maintenance  personnel  reseat  the  SRU  in 
the  LRU. 

Such  a  total  development  package  could  triple  the  mean  time 
between  indicated  faults  (from  four  hours  to  13  hours)  and  also  reduce 
the  maintenance  workload  by  50  percent.  In  total,  this  development 
package  (excluding  Priority  3  items)  would  cost  $200  million,  but  it  is 
estimated  that  much  of  this  cost  could  be  offset  by  the  decreased  need 
for  maintenance. 

Table  3  summarizes  the  benefits  and  costs  of  implementing  recom¬ 
mended  improvements  for  both  the  F-15  and  F-16  radars. 


Table  3 

BENEFITS  AND  COSTS  OF  IMPLEMENTING  RECOMMENDED  PRIORITY  1 
AND  PRIORITY  2  IMPROVEMENTS 


Radar 

Mean  Time 
Between 
confirmed 
Failure8 
(hours) 

Fault 

Removal 

Efficiency^ 

(%) 

Additional 

Cost 

(million 

1984  $) 

APG  63  on  the  F-15  C/D 
Observed  during  1984 

19 

21 

Projected  with 

25 

50 

200 

improvements 

APG  66  on  the  F-16  A/B 
Observed  during  1984 

82 

7 

Projected  with 

100 

55 

250 

improvements 

*Mean  Time  Between  confirmed  Failure  (MTBF)  is  based  on  shop  confirmation 
of  failure. 

bA  fault  removal  efficiency  of  21  percent  means  that  21  percent  of  the  flights 
with  a  radar  fault  indicated  ended  up  with  the  support  process  removing  faulty 
equipment  from  the  airplane. 
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The  benefits  to  be  gained  in  spending  the  $200  million  to  improve 
the  F-15  C/D  radar  also  appear  to  be  a  substantial  rise  in  fault  removal 
efficiency,  from  a  current  rate  of  about  20  percent  to  an  improved  rate 
of  around  50  percent. 

This  increase  in  fault  removal  efficiency,  combined  with  a  lower 
MTBF  than  the  F-16  radar,  would  yield  a  major  increase  in  the 
number  of  fault-free  flights.  Although  the  precise  magnitude  of  the 
increase  is  unclear,  again  because  we  don’t  know  how  many  flights 
were  flown  with  faults  that  generated  no  symptoms,  we  can  again  esti¬ 
mate  the  potential  magnitude  by  assuming  a  ratio  for  the  total  number 
of  faulty  flights  to  the  number  of  flights  with  symptoms.  Assuming 
that  the  number  of  fault-plagued  flights  is  double  the  number  with 
indicated  faults,  then  the  incorporation  of  the  improvement  package 
would  increase  fault-free  flights  from  34  to  79  percent.11 

From  the  specific  R&M  improvement  packages  for  the  F-15  and 
F-16  radars,  we  next  turn  to  lessons  that  these  packages  suggest.  Each 
has  a  bearing  on  building  the  consensus  needed  to  support  timely 
implementation  of  important  improvements. 

•  Important  benefits  to  be  derived  from  improved  R&M  include 
matters  other  than  reduction  in  costs  of  operations  and  support. 
Principal  among  these  are  increased  availability  of  the  full 
measure  of  the  equipment’s  designed  capabilities.  Although  the 
true  economic  benefit  of  such  improvement  is  difficult  to  gauge 
in  quantitative  terms,  understanding  of  such  benefit  is  crucial 
to  making  sound  choices  about  product  improvement. 

•  A  key  to  such  increased  availability  is  increased  efficiency  in 
fault  removal. 

•  Increasing  fault  removal  for  an  already  fielded  system  can  be 
very  costly  when  hardware  must  be  retrofitted. 

•  Causes  of  R&M  problems  may  lie  in  many  places:  the  airborne 
equipment,  its  built-in  tests,  its  support  equipment,  its  support 
procedures,  and/or  debriefing  practices  of  flight  crews. 

•  A  cohesive  program  to  mature  weapon  system  R&M  may 
require  implementation  actions  by  many  organizations  within 
the  Air  Force. 

Building  a  consensus  for  necessary  and  timely  action  on  R&M 
improvements  requires  not  only  sound  information,  but  also  a  special 
kind  of  cooperation  by  involved  organizations. 

"An  assumption  of  1.5  faulty  flights  for  each  flight  with  a  fault  indicated  means  that 
the  incorporation  of  the  improvement  package  would  increase  fault-free  flights  from  50 
to  85  percent. 
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No  Continuity  in  Weapon  System  Program  Management 
Responsibility.  Achieving  such  cooperation  is  complicated  by  the 
practice  of  transferring  program  management  responsibility  (PMRT). 

In  the  case  of  avionics,  engineering  management  responsibilities  are 
transferred  from  the  SPO  to  the  Air  Force  Logistics  Command  (AFLC) 
long  before  R&M  maturation  has  occurred.  This  generally  occurs  dur¬ 
ing  the  equipment’s  early  operational  life,  when  engineers  assigned  to 
the  SPO  are  only  beginning  to  have  an  opportunity  to  identify  the 
equipment’s  strengths  and  weaknesses. 

Once  the  AFLC  System  Program  Manager  assumes  responsibility,  a 
new  and  much  smaller  group  of  Air  Force  engineers  becomes  responsi¬ 
ble  for  the  equipment’s  further  maturation.  These  engineers  must  pro¬ 
duce  entirely  new  contracting  documents  to  begin  any  redesign  work. 
This  contracting  procedure  is  far  more  burdensome  than  the  one  that 
the  SPO  follows  to  do  the  same  sorts  of  design  changes  before  PMRT. 
Thus,  any  improvements  in  equipment  maturation  slow  considerably 
after  PMRT.  With  an  impending  transfer  of  R&M  engineering 
responsibilities,  it  is  also  difficult  to  keep  the  development  organization 
interested  in  improving  R&M,  especially  when  routinely  collected 
statistics  reveal  no  great  need  for  such  improvements. 

To  illustrate  the  effect  of  these  difficulties,  consider  the  case  of  the 
LPRF.  In  part  because  PMRT  has  occurred,  and  in  part  because  a 
consensus  has  been  slow  to  materialize,  the  contract  for  the  develop¬ 
ment  of  a  new  LPRF  unit  cannot  be  let  until  late  1989.  Figure  18 
shows  that  this  is  almost  four  years  after  engineering  efforts  analyzed 
the  problem  and  recommended  redesigning  the  LPRF  unit. 

Poor  Coordination  of  Management  of  Weapon  System  after 
PMRT.  Following  PMRT,  there  is  another  complication.  Entirely 
different  Air  Logistics  Centers  may  be  responsible  for  the  airborne 
equipment  and  the  support  equipment.  Although  in  theory  these  orga¬ 
nizations  work  together,  in  practice  the  weapon  system  no  longer 
enjoys  the  single  source  of  management  attention  that  the  SPO  once 
could  devote  to  it.  Such  splitting  of  management  attention  after 
PMRT  poses  special  challenges  to  a  timely  implementation  of  a 
cohesive  program  of  improvements  such  as  the  ones  illustrated  in 
Tables  1  and  2. 


STRENGTHENING  THE  PRODUCT 
IMPROVEMENT  PROCESS 

To  strengthen  the  responsiveness  of  the  product  improvement  pro¬ 
cess  to  the  causes  of  a  weapon  system’s  dominant  R&M  problems  we 
propose  a  three-part  strategy  that  includes  improving  ih»  flow  of 
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information  from  the  field,  increasing  field  engineering,  and  expediting 
important  improvements. 

Improving  Information  from  the  Field 

By  applying  the  proposed  new  view  of  R&M  to  data  already  being 
acquired  by  the  Air  Force’s  existing  MDC  system,  the  Air  Force  can 
better  identify  areas  where  further  R&M  investigation  is  most 
needed.12  Monitoring  and  analyzing  information  from  the  previously 
proposed  tracking  of  avionics  performance  by  equipment  serial  number 
and  determining  why  the  routine  support  process  fails  to  correct  prob¬ 
lems,  would  make  the  product  improvement  process  more  aware  of  the 
dominant  R&M  problems  being  experienced  in  the  field. 

Increasing  Field  Engineering 

A  rational  approach  to  R&M  management  must  include  arrange¬ 
ments  for  addressing  the  reality  that  following  IOC  more  work  will 
have  to  be  done  on  any  system — especially  sophisticated  systems  such 
as  radars — to  raise  the  maturity  of  their  R&M  characteristics  to  an 
acceptable  level.  One  way  to  do  this  is  to  get  equipment  contractor 
engineers  more  involved  in  understanding  R&M  from  the  viewpoint  of 
the  operators  and  the  maintainers  in  the  field.  A  capability  to  do 
performance-oriented  tracking  of  equipment  after  repairs  have  been 
completed  will  help,  but  an  actual  presence  in  the  field  will  be  required 
to  identify  the  causes  of  major  problems  in  the  more  complex  subsys¬ 
tems.  This  will  require  more  special  data  collection  activities  such  as 
the  one  that  emphasized  the  F-15  and  F-16  radars. 

Expediting  Important  Improvements 

Effective  product  improvement  must  also  be  timely  product  improve¬ 
ment.  Special  data  collection  and  analysis  efforts  should  be  scheduled 
to  begin  immediately  after  the  Air  Force  has  a  few  squadrons  at  IOC. 
That  change  alone  would  have  saved  about  four  years  from  the  time 
line  in  Fig.  18. 

The  actual  development  of  the  most  critically  needed  improvements 
should  begin  right  after  the  data  collection  and  analysis  phase.  In  the 
F-16  radar  case  that  would  have  saved  an  additional  four  years.  To  do 
this,  the  Air  Force  would  need  to  preprovision  funding  so  that  the 
implementing  improvements  phase  can  be  scheduled  immediately 

12Petruschell,  Smith,  and  Kirkwood,  1987. 
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following  analyses  and  decisions  about  recommended  improvements. 
The  Air  Force  would  also  need  to  establish  policies  and  procedures  for 
arriving  at  consensus  to  expedite  important  improvements.  If  such 
funding  and  programmatic  provisions  had  been  made  for  the  F-16  A/B 
Radar  R&M  Improvement  Program,  the  retrofitting  of  a  new  LPRF 
would  already  be  completed,  instead  of  having  to  wait  till  1994.  But 
even  more,  there  would  be  fewer  items  to  retrofit  in  the  F-16  radar 
because  the  development  effort  would  have  broken  into  the  production 
line  before  production  was  concluded  on  the  LPRF  LRU. 

By  adopting  such  immediate  data  collection  and  development  for 
critical  improvement  i,  the  Air  Force  could  initiate  much  of  the  most 
important  R&M  maturation  work  before  PMRT,  thus  avoiding  further 
delays. 


SUMMARY 

By  improving  the  amount  and  source  of  field  information  with 
better  collection  efforts  and  in-depth  field  engineering  investigation, 
the  Air  Force  gains  a  very  different  and  extremely  useful  perspective 
on  the  R&M  needs  of  weapon  systems.  Knowledge  of  the  most  serious 
R&M  problems  with  vital  equipment  and  associated  support  resources 
increases  immensely.  Better  knowledge  coupled  with  improved  pro¬ 
cedures  for  building  a  consensus  on  what  needs  to  be  done  for  specific 
weapon  systems  should  lead  to  wiser  and  more  beneficial  investments 
in  product  improvement,  thus  enhancing  vital  R&M  characteristics. 


V.  THE  ACQUISITION  PROCESS 


The  new  view  of  weapon  system  R&M  emphasizes  two  R&M  charac¬ 
teristics: 

•  Fault  initiation, 

•  Fault  removal. 

Although  the  first  is  determined  exclusively  by  the  airborne  equip¬ 
ment,1  the  second  is  influenced  by  a  combination  of  that  equipment 
and  the  associated  array  of  ground  support  equipment  used  by  techni¬ 
cians  on  the  flight  line,  in  the  air  base  shops,  and  at  the  depot’s  repair 
facilities.  Thus,  to  fulfill  responsibilities  for  both  characteristics,  the 
weapon  system  acquisition  process  needs  to  research,  develop,  and  pro¬ 
cure 

1.  Airborne  equipment  that  will  dependably  deliver  the  Air 
Force’s  specified  levels  of  performance  when  that  equipment  is 
operated  and  maintained  by  people  of  normal  skill  and  in 
accordance  with  Air  Force  approved  technical  data, 

2.  Weapon-system-peculiar  ground  support  equipment  needed  for 
routine  support  of  the  designed  capabilities  of  the  airborne 
equipment, 

3  Technical  data  and  any  special  equipment  needed  to  support 
the  correction  of  any  faults  not  resolved  by  the  routine  sup¬ 
port  process.2 

Such  a  broad  view  of  weapon  system  acquisition  responsibilities  is  key 
to  the  notion  that  weapon  system  maintainability  needs  to  reflect  the 
ease  with  which  the  complete  support  process  (from  flight  line  through 
depot)  can  restore  the  designed  capabilities  of  airborne  equipment. 


WEAKNESSES  IN  THE  WEAPON  SYSTEM 
ACQUISITION  PROCESS 

Our  review  of  the  ability  of  the  current  acquisition  process  to  pro¬ 
vide  for  both  R&M  characteristics  has  identified  three  weaknesses: 


'We  always  use  the  term  equipment  to  refer  to  hardware  and  any  associated  software. 
2Recall  the  need  for  test  translation  dictionaries  and  special  equipment  for  the  depot 
repair  facilities  to  fix  bad  actor  equipment  (Sec.  III). 
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•  Failure  to  use  a  meaningful  set  of  management  measures  for 
R&M, 

•  Lack  of  a  process  for  setting  rationally  based  R&M  goals  for 
future  equipment, 

•  No  strong  assurances  that  needed  levels  of  R&M  will  be 
delivered. 

Failure  to  Use  a  Meaningful  Set  of  Management 
Measures  for  R&M 

Especially  within  the  acquisition  process,  the  traditional  view  of 
R&M  assumes  that  equipment  is  broken  or  it’s  fixed  and  it’s  easy  to 
identify  when  it  is  broken.  Such  a  view  leads  to  relatively  simple 
management  measures  for  characterizing  R&M:  mean  time  between 
failure  (MTBF),  mean  time  to  repair  (MTTR),  and  fully  mission  capa¬ 
ble  (FMC)  rate.  Such  a  set  of  measures  provides  only  a  partial  picture 
of  the  R&M  situation.  The  lack  of  situational  awareness  greatly  weak¬ 
ens  the  ability  of  the  acquisition  process  to  respond  fully  to  R&M 
needs. 

An  effort  to  improve  R&M  situational  awareness  must  cope  with 
two  challenges.  First,  R&M  is  a  quality  that  has  many  dimensions. 
Second,  precise  characterization  of  this  quality  becomes  more  difficult 
as  the  complexity  of  the  equipment  and  its  support  process  grows. 
Each  of  these  realities  is  especially  applicable  to  modern  military 
avionics. 

For  such  equipment  the  two  main  dimensions  of  interest  are  the 
mean  time  between  initiation  of  new  faults  and  the  efficiency  of  the 
maintenance  process  in  removing  faults.  The  usual  practice  is  to  esti¬ 
mate  MTBF  by  calculating  the  mean  time  between  shop  confirmed 
failures;  that  usually  provides  a  plausible  estimate  for  the  mean  time 
between  initiation  of  new  faults.3  To  complement  the  MTBF  parame¬ 
ter,  the  Air  Force  needs  a  parameter  that  reflects  the  efficiency  of 
maintenance  technicians  in  fully  restoring  the  designed  performance  of 
a  system  once  a  fault  has  developed.  Lacking  such  a  measure,  the 
acquisition  process  is  failing  to  attract  management  attention  to  the 
more  serious  maintainability  problems. 

Lack  of  a  Process  for  Setting  Rationally  Based  R&M  Goals 

Even  with  the  measures  currently  used  in  the  acquisition  process, 
the  Air  Force  lacks  a  rationally  based  process  for  setting  R&M  goals 
for  future  equipment.  Although  broad  goals,  such  as  direction  to 

3This  assumes  that  the  equipment  and  its  support  process  have  achieved  a  state  of 
equilibrium  where  the  rate  at  which  the  shop  is  confirming  (and  removing)  faults  exactly 
matches  the  rate  at  which  new  faults  are  being  generated. 
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double  weapon  system  reliability  and  halve  maintenance,  convey  a  mes¬ 
sage  that  the  Air  Force  is  serious  about  reliability  and  reduced  mainte¬ 
nance,  more  specific  guidance  needs  to  be  established  for  equipment 
developers. 

Consider  the  idea  of  simply  doubling  the  reliability  of  everything  in 
the  weapon  system.  First,  reliability  is  not  a  free  good;  it  is  achieved 
through  investment  of  time  and  resources.  Moreover,  with  most  equip¬ 
ment  and  parts  the  cost  of  increasing  reliability  becomes  more  expen¬ 
sive  as  the  level  of  reliability  rises.  Second,  the  achieved  level  of  reli¬ 
ability  is  much  like  the  strength  of  a  chain;  it  is  determined  by  the 
strength  of  its  weakest  link. 

In  many  cases,  a  weapon  system’s  overall  reliability  will  be  dom¬ 
inated  by  the  reliability  of  about  one-fourth  or  fewer  of  its  subsystems. 
Each  subsystem  likewise  has  a  reliability  that  may  be  dominated  by 
perhaps  only  one-fourth  of  its  major  assemblies.  Each  such  major 
assembly’s  reliability  may  be  dominated  by  only  about  one-fourth  of  its 
subassemblies.  And  finally,  each  such  subassembly’s  reliability  may  be 
dominated  by  only  about  one-fourth  of  its  parts.  Thus,  reliability  of  a 
weapon  system  will  be  dominated  by  a  very,  very  small  portion  of  its 
total  parts.  In  our  hypothetical  example,  there  would  be  only  about 
four  parts  in  one  thousand4  that  would  dominate  overall  R&M.  The 
challenge  for  the  acquisition  process  is  to  direct  attention  to  those 
areas  that  provide  the  greatest  leverage  in  terms  of  cost  beneficial 
improvements  to  R&M. 

Reliability  goals,  at  all  levels  of  assembly,  need  to  be  carefully  estab¬ 
lished  in  view  of  not  only  the  prospective  contribution  to  overall 
weapon  system  reliability,  but  also  the  differential  costs  of  achieving 
reliability  goals  for  different  types  of  equipment.  A  rationally  based 
goal  setting  process  also  needs  to  address  the  differential  nature  of  the 
benefits  to  be  derived  from  alternative  levels  of  reliability  for  specific 
types  of  equipment.  Consider  an  analysis  of  what  improved  reliability 
of  the  F-16  A/B  weapon  system  could  have  yielded  in  several  dimen¬ 
sions  of  different  benefit:6 

•  Reductions  in  base-level  maintenance  manpower 

•  Reductions  in  certain  engine  and  spares  costs 

•  Improvements  in  squadron  deployment  flexibility. 


*.25  x  .26  x  .26  x  .26  -  .004. 

5For  a  full  description  of  this  potential  benefit  analysis,  see  Abell,  Kirkwood,  and 
Petruschell,  1988.  Although  the  actual  amount  of  estimated  benefits  is  substantial,  it 
was  beyond  the  scope  of  the  analysis  to  estimate  the  costs  or  other  resources  that  would 
have  been  required  to  achieve  the  postulated  reliability  improvements. 
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To  simplify  the  calculations,  we  assumed  that  the  mean  time  between 
maintenance  request  (MTBMR)  would  also  quadruple,  as  would  the 
mean  time  between  demand  (MTBD)  for  parts.  These  assumptions  are 
reasonable  approximations  to  the  extent  that  maintainability  charac¬ 
teristics  do  not  change  to  any  appreciable  extent.6 

Maintenance  Manning.  A  twofold  improvement  in  the  overall 
reliability  of  the  aircraft — i.e.,  a  50  percent  reduction  in  the  rate  of 
maintenance  actions  on  all  components  of  the  aircraft — could  reduce 
maintenance  manpower  requirements  an  estimated  9  percent,  given  the 
same  level  of  wartime  flying  activity.7 

The  manpower  savings  occur  almost  entirely  in  the  component 
repair  squadron.  The  requirements  for  avionics  technicians  could  be 
reduced  by  about  a  fourth  and  for  propulsion  technicians  by  about  a 
third,  accompanied  by  more  modest  reductions  in  other  skills.  Includ¬ 
ing  training  costs,  these  reductions  would  represent  an  annual  savings 
of  about  $5  million  per  72-PAA  wing. 

Spares  Cost.  The  effects  of  improved  reliability  (as  reflected  by 
component  removal  rates)  on  the  investment  costs  of  spare  engines  and 
engine  modules,  and  the  procurement,  depot-level  repair,  and  condem¬ 
nation  costs  of  recoverable  aircraft  spares  for  the  F-16  A/B  program 
were  also  estimated  in  this  work.  A  twofold  reliability  improvement  in 
fire-control  and  propulsion  system  components  alone  (as  reflected  by  a 
50  percent  reduction  in  component  removal  rates),  given  a  constant  80 
percent  aircraft  availability  goal,  would  have  yielded  an  estimated  cost 
avoidance  of  $1.2  billion  in  constant,  undiscounted  FY84  dollars  in 
these  resource  categories  over  the  13-year  period  from  1978  through 
1990. 

Deployability.  Improving  reliability  also  decreases  the  number  of 
personnel  and  the  tonnage  of  spares  and  equipment  required  to  support 
tactical  deployments.  The  postulated  twofold  reliability  improvement 
for  a  24-PAA  F-16  A/B  squadron  would  reduce  its  burden  of  bare-base 
deployment  support  during  the  first  30  days  of  a  deployment  by  an 
estimated  40  tons.  Although  seemingly  large  in  absolute  terms,  it  is 
only  about  5  percent  of  the  total  tonnage  required,  much  of  which 
comprises  tractors,  trailers,  and  loaders  for  handling  munitions  and 
towing  aircraft.  Thus,  reliability  does  not  dramatically  effect 

8In  the  previous  section  we  addressed  the  possibility  of  major  changes  in  maintain¬ 
ability  that  also  could  cause  substantial  changes  in  both  MTBMR  and  MTBD.  However, 
regardless  of  what  mixture  of  maintainability  advances  and  reliability  improvements 
were  to  achieve  a  fourfold  improvement  in  MTBMR  and  MTBD,  the  derived  benefits 
would  remain  the  same. 

7Altematively,  the  reduction  in  maintenance  manning  described  could  be  traded  for 
an  estimated  17  percent  increase  in  wartime  sortie  generation  capability  with  current 
manning  levels. 
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deployment  support  requirements.8  Figure  20  summarizes  the  benefits 
for  two  cases: 

•  A  doubling  in  reliability  (results  designated  by  the  “2x”), 

•  A  quadrupling  in  reliability  (results  designated  by  the  “2x”). 

The  results  designated  by  the  “lx”  reflect  the  weapon  system’s  current 
level  of  reliability.  The  findings  reported  in  Abell,  1988,  have  been 
adjusted  to  630  aircraft,  four  wings  deployed,  a  15-year  life  cycle,  and 
1986  dollars. 

In  percentages,  the  manpower  (air  base  maintenance)  benefits  may 
not  look  striking — a  fourfold  improvement  in  mean  time  between 
maintenance  requests  yields  about  a  15  percent  reduction  in  manpower. 
But  consider  that  much  of  these  savings  of  about  2,000  people  world¬ 
wide  would  be  in  high-skilled  avionics  and  propulsion  specialties. 


Air  base  Spares  and 

Airlift  maintenance  depot  repair 


Fig.  20 — Summary  of  benefits  from  study  of  improved  reliability  on 
F-16  A/B  weapon  systems 


®The  airlift  (deployment)  benefit  for  the  F-16  is  modest  because  during  the  first  30 
days  of  a  deployment  the  F-16  does  not  deploy  an  Avionics  Intermediate  Shop  (A1S).  A 
similar  case  study  for  the  F- 15,  whose  AIS  is  deployed  with  the  aircraft,  would  yield  a 
much  greater  airlift  savings. 
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Evidently  the  area  of  the  greatest  benefits  from  improved  reliability 
is  the  costs  of  spares  and  depot  repair  (excluding  base  level  mainte¬ 
nance  repair  cost).  Doubling  the  reliability  (in  terms  of  mean  time 
between  demand  for  parts)  yields  a  $1.2  billion  savings  over  the  15-year 
life  cycle  of  the  systems.  A  fourfold  improvement  yields  yet  another 
$600  million  in  reductions.  Because  the  results  show  a  diminishing 
return  to  further  increases  in  reliability,  no  more  than  a  fourfold 
improvement  in  the  overall  mean  time  between  demand  for  parts  for 
the  next  fighter  airplane  may  be  a  reasonable  requirement. 

Individual  analyses  for  each  subsystem  would  need  to  establish  pre¬ 
cise  goals  for  individual  types  of  subsystems.  Such  analyses  must  con¬ 
sider  the  prospective  cost  of  achieving  alternative  levels  of  reliability. 
Because  such  costs  depend  on  many  factors  peculiar  to  the  technologies 
used  in  a  design,  as  well  as  the  design  itself  and  conditions  of  equip¬ 
ment  use,  we  can  not  make  a  general  forecast  of  tl  cost  of  alternative 
levels  of  reliability. 

Appropriate  R&M  management  measure(s)  for  maintainability  need 
to  be  addressed  in  addition  to  reliability. 

No  Strong  Assurances  that  Needed  Levels  of  R&M 
Will  Be  Delivered 

Once  appropriate  R&M  goals  are  established,  the  next  challenge  for 
the  acquisition  process  is  to  provide  strong  assurance  that  the  needed 
reliability  and  maintainability  characteristics  will  be  delivered. 
Because  of  the  complexity  of  aircraft  weapon  systems  and  their  associ¬ 
ated  support  systems  and  the  urgency  normally  associated  with  their 
development  and  initial  fielding,  it  is  only  prudent  to  base  our 
approach  to  acquisition  on  the  view  that: 

Following  initial  introduction  to  operation,  some  significant 
amount  of  product  improvement  will  be  required  to  mature 
R&M  characteristics  to  needed  levels. 

The  Air  Force’s  ability  to  assure  that  needed  levels  of  R&M  will  be 
delivered  is  determined  by  the  strength  of  the  product  improvement 
efforts  that  can  be  brought  to  bear  in  areas  of  deficiencies.  Needed 
product  improvement  activity  is  lacking  in  the  aircraft  weapon  system 
acquisition  process,  notwithstanding  two  classes  of  efforts  by  the  Air 
Force: 

1.  Warranties  and 

2.  Government  funded  R&M  improvement  programs. 
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Warranties.9  Although  warranties  have  long  been  a  common  part 
of  commercial  business  practice,  often  as  an  element  of  price  competi¬ 
tion,  until  recently  they  were  applied  only  occasionally  to  military 
equipment,  mainly  avionics  and  other  subsystems.  The  services 
expressed  satisfaction  with  some  applications  and  cited  problems  with 
others.  Then  in  1983  Congress  passed  the  Defense  Appropriations  Act 
for  1984,  since  modified  by  the  Defense  Authorization  Act  of  1985, 
requiring  the  armed  services  to  obtain  warranties  on  all  major  buys  of 
weapon  systems. 

The  Congressional  intent  seems  to  be  a  combination  of  desires  based 
upon  pragmatic  considerations  about  the  acquisition  process.  Major 
aircraft  weapon  system  acquisition  programs  usually  contain  large  ele¬ 
ments  of  uncertainty  and  urgency,10  once  started  they  are  difficult  to 
abandon,  and  major  weapon  deliveries,  even  if  very  short  of  certain 
specifications,  are  difficult  to  reject.  Newly  developed  weapon  systems 
generally  represent  such  substantial  advances  on  the  equipment  they 
are  to  replace  that  the  government  is  ill-disposed  to  writing  off  years  of 
development  by  rejecting  delivery.  The  services  recognize,  moreover, 
that  the  road  to  fuller  compliance  with  the  mission-essential  aspects  of 
the  performance  specifications  often  requires  working  with  the 
contractor’s  engineering  staff  rather  than  becoming  embroiled  in  legal 
disputes. 

The  Congress,  growing  impatient  with  recent  acquisition  program 
outcomes,  seems  to  want  a  better  track  record  in  terms  of  the  match 
between  the  performance  specifications  promised  during  the  bidding 
process  and  the  performance  eventually  measured  in  the  field.  War¬ 
ranties  seem  to  offer  an  opportunity  for  remedies  that  can  be  tied  pro¬ 
portionally  to  the  shortfalls  in  essential  performance.  Congress  seems 
to  see  requiring  warranties  as  a  way  to  instill  more  discipline  in  the 
relationships  between  the  services  and  their  contractors. 

Some  view  warranties  as  providing  an  opportunity  to  insure  acquisi¬ 
tion  program  outcomes  because  the  contractor  is  obliged  to  accept 
prescribed  risks  that  empower  the  government  to  assess  blame  and 
extract  damages  in  a  predetermined  and  agreed  upon  manner.  Major 
concerns  about  such  a  strong  view  include  some  specific  to  R&M  in 
addition  to  some  of  a  general  nature: 

•  First,  pragmatic  considerations  still  limit  the  economic  value  of 
the  maximum  damages  that  the  government  can  extract,  no 
matter  how  great  the  economic  cost  of  the  damage. 

^For  a  more  detailed  discussion  see  Stucker  and  Smith,  1987. 

10Most  of  the  major  weapons  bought  by  the  military  are  very  advanced,  complex, 
risky  products  that  do  not  have  counterparts  in  the  commerc.al  world. 
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•  Second,  there  are  serious  questions  about  the  extent  to  which 
the  government  can  totally  shift  risk  to  the  contractor.  The 
government  shares  a  role  in  approving  design  and  setting  forth 
specifications11  that  subsequently  may  prove  either  beyond  the 
state  of  the  art  or  vulnerable  to  some  other  supervening  impos¬ 
sibility12  that  interferes  with  the  contractor’s  ability  to 
discharge  his  responsibilities.  Either  event  lays  grounds  for 
legal  disputes  about  the  validity  of  a  contract. 

•  Third,  even  if  pragmatic  and  legal  considerations  do  not  interfere, 
economists  will  caution  that  it  is  rarely  to  the  government’s 
advantage  to  buy  insurance  from  a  private  firm.13  Most  analysts 
believe  the  government  should  usually  self-insure:  It  has  more 
resources  than  any  firm  and  a  better  ability  to  survive  losses,  it 
has  more  projects  under  way  at  a.iy  given  time  and  can  better 
spread  large  risks,  and  it  is  less  risk-averse  than  typical  profit- 
motivated  firms. 

These  general  concerns  flow  from  the  inescapable  reality  that  major 
weapon  systems  differ  from  commercial  products  in  fundamental  ways 
that  will  effect  the  manner  in  which  warranties  should  be  applied  to 
military  equipment.  For  example,  the  primary  objective  of  a  weapon 
system  warranty,  as  distinct  from  a  commercial  warranty,  should  be 

To  mature  the  weapon  system  to  a  point  where  there  is  an 
acceptable  probability  that  the  system  will  deliver  the  full 
measure  of  its  designed  capability  whenever  called  upon  in 
combat. 

When  a  fighter  pilot  is  assigned  an  aircraft  to  take  into  combat,  what 
he  needs  is  confidence  that  his  afterburner,  radar,  and  other  essential 
equipment  are  prepared  to  deliver  the  full  measure  of  their  designed 
capabilities  during  combat. 


''The  design  and  development  of  most  major  military  acquisitions  are  typically  super¬ 
vised  and  controlled  by  the  services:  The  product  is  not  uniquely  the  responsibility  of 
the  firm  that  develops  and  produces  it;  and  that  firm  is  usually  not  in  a  position  to  take 
full  responsibility  for  the  design. 

12For  example,  military  products  are  frequently  ordered  into  production  long  before 
their  test  programs  are  completed,  and  certainly  long  before  their  R&M  characteristics 
can  be  fully  validated. 

13Although  it  may  be  appropriate  to  view  warranties  for  commercial  products  as  a 
form  of  insurance,  there  are  many  different  views  of  warranties  in  the 
govemment/contractor  context.  One  view  holds  that  it  is  naive  to  think  of  warranties  in 
this  context  as  insurance  when  the  main  purpose  of  the  warranty  is  to  try  to  get  the  con¬ 
tractor  to  bid  honestly. 
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To  see  whether  warranties  might  provide  assurance  for  achieving 
such  an  objective,  we  examined  two  sets  of  warranties: 

•  One  set  written  before  the  new  laws  were  passed 

•  The  second  set  written  since  January  1,  1985. 

Although  our  examination  of  the  older  warranties  was  limited  by  a 
dearth  of  available  information,  reading  of  program  documents  and  dis¬ 
cussions  with  government  and  industry  officials  suggest  that  four  fac¬ 
tors  may  be  important  to  the  ability  of  such  warranties  to  contribute  to 
R&M  objectives:14 

•  Specific,  measurable,  pertinent  objectives, 

•  Explicit,  unambiguous  remedies, 

•  Explicit  duties  for  the  contractor  and  government, 

•  Reasonable  prices  and  expectations. 

At  this  time  we  cannot  say  that  these  factors  insure  favorable  out¬ 
comes,  or  even  that  they  are  usually  associated  with  them.  But  they 
were  important  to  warranty  programs  that  people  have  viewed  favor¬ 
ably,  and  their  absence  was  frequently  mentioned  as  a  problem  on  the 
more  troublesome  programs. 

Examination  of  the  set  written  to  comply  with  the  1985  law  revealed 
some  interesting  contrasts  with  the  earlier  warranties.  In  particular, 
two  major  attributes  that  contributed  to  the  success  of  earlier  warran¬ 
ties  are  missing  from  many  of  the  newer  ones:  the  simple  definition  of 
measurable  objectives  and  the  precise,  prespecified  remedies  expected 
of  the  contractor.16  We  are  concerned  that  such  nonspecificity  can 
easily  lead  to  misunderstandings  and  possibly  to  threats  of  nonperfor¬ 
mance  and  litigation. 

Because  warranties  are  being  applied  in  so  many  diverse  forms,  the 
systematic  acquisition  of  information  about  these  different  experiences 
is  essential  to  building  a  body  of  knowledge  upon  which  future  warran¬ 
ties  might  more  fruitfully  be  framed  and  applied  in  areas  where  they 
have  the  most  cost  beneficial  effect.  Unfortunately,  we  find  that  the 
services  are  spending  a  lot  of  money  and  effort  specifying,  negotiating, 
implementing,  and  enforcing  warranties,  without  much  information  on 
whether  it  is  all  worthwhile.  Meanwhile,  although  data  and  contrac¬ 
tual  language  are  being  gathered  sporadically  by  various  agencies,  a 
systematic  and  coordinated  process  has  not  yet  been  established  for 
acquiring  data  essential  to  evaluate  the  effectiveness  of  individual  war¬ 
ranty  programs. 

14See  Stacker  end  Smith,  1987,  Sec.  III. 

18Stacker  and  Smith,  1987. 
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Lacking  solid  evidence  of  a  sufficient  connection  between  warranties 
and  achieved  levels  of  R&M,  we  would  be  imprudent  to  rely  solely  on 
warranties  to  mature  a  complex  aircraft  weapon  system  to  a  point 
where  there  is  an  acceptable  probability  that  the  system  will  deliver  the 
full  measure  of  its  designed  capability  whenever  called  upon  in  combat. 

Government  Funded  Product  Improvement  Programs.  Not 
only  have  warranties  fallen  short  of  providing  strong  assurances  that 
development  programs  will  deliver  needed  levels  of  R&M,  but  the  usual 
government  funded  product  improvement  programs  have  also  fallen 
short.  Even  the  special  F-15/F-16  Radar  R&M  Improvement  Program 
has  yielded,  at  best,  a  slow  response  to  the  most  needed  improvements. 


STRATEGY  FOR  IMPROVING  THE  WEAPON 
SYSTEM  ACQUISITION  PROCESS 

To  strengthen  the  ability  of  the  acquisition  process  to  meet  its 
responsibilities  to  deliver  weapon  systems  with  needed  R&M  charac¬ 
teristics,  we  propose  a  strategy  that  includes  expanding  awareness  of 
R&M  deficiencies,  emphasizing  fault  removal  efficiency,  accelerating 
maturation  of  avionics,  and  reorganizing  avionics  development. 

Expanding  Awareness  of  R&M  Deficiencies 

To  improve  the  R&M  record  of  the  acquisition  process,  the  govern¬ 
ment  and  industry  organizati-  ns  that  are  responsible  for  the  development 
of  new  equipment  need  to  expand  their  awareness  of  the  dominant  R&M 
deficiencies  in  currently  fielded  equipment.  Implementation  of  the  pro¬ 
posals  to  strengthen  the  support  process  and  the  product  improvement 
process  will  highlight  such  deficiencies  and  their  root  causes.  Thus, 
efforts  to  improve  the  acquisition  process  will  benefit  from  improvements 
to  these  other  processes.  An  improved  acquisition  process  will  in  turn 
lighten  the  burdens  that  now  must  be  borne  by  these  other  processes. 

Emphasizing  Fault  Removal  Efficiency 

Improved  awareness  of  the  dominant  problems  is  only  an  initial  step 
that  needs  to  be  followed  by  a  greater  emphasis  on  fault  removal  effi¬ 
ciency  in  new  developments.  We  propose  that  the  Air  Force  adopt 
fault  removal  efficiency  in  conjunction  with  the  MTBF  parameter  to 
provide  a  comprehensive  capability  to  view  the  overall  R&M  situation 
for  complex  subsystems  in  aircraft  weapon  systems. 


66 


We  base  this  proposal  on  the  following  goals: 

•  Provide  a  comprehensive  overview  of  the  R&M  situation  at  the 
subsystem  level, 

•  Reflect  the  full  range  of  problems  that  arise  in  identifying  faults 
and  isolating  their  causes, 

•  Account  for  all  flights  with  indications  of  faulty  subsystem 
operation. 

To  achieve  such  thorough  accounting,  pilots  and  technicians  need  to 
report  all  indications  of  situations  where  either  they  or  the  BIT  per¬ 
ceive  a  subsystem’s  performance  departing  from  designed  capabilities. 
Such  full  reporting  of  symptoms  is  absolutely  essential.16  Such 
thorough  accounting  for  all  perceived  indications  of  faulty  operations  is 
a  complete  change  from  the  comfortable  traditional  view  that  a  prob¬ 
lem  not  found  on  the  ground  did  not  exist  in  the  air.17 

Approach  to  Estimation.  The  following  method  for  estimating 
fault  removal  efficiency  is  consistent  with  the  noted  goals: 


Fault 

Removal 

Efficiency 


MTBI 

-  x  100% 

MTBF 


where 

MTBI  -  Mean  flight  time  between  flights  with  indica¬ 
tion^)  of  faulty  operation  of  the  avionics  sub¬ 
system. 

MTBF  =  Mean  flight  time  between  flights  that  resulted  in 
shop  confirmation  of  a  failure  of  the  avionics 
subsystem. 

Critical  Concerns.  To  arrive  at  a  meaningful  MTBI,  pilcts  not 
only  need  to  report  all  indications  of  suspected  faults,  but  maintenance 
personnel  need  to  maintain  a  historical  record  of  such  pilot  reports  for 
the  critical  subsystems  on  each  aircraft.18 

16It  also  depends  upon  the  pilot’s  ability  to  recognize  degradations  in  performance. 
Such  ability  can  be  enhanced  by  making  pilots  aware  of  criteria  that  they  should  apply 
and  special  tactics  they  can  employ  to  verify  the  performance  of  their  equipment. 

17Such  a  philosophy  becomes  increasingly  inappropriate  as  the  nature  of  avionics 
faults  shifts  toward  ones  where  symptoms  are  situation-dependent.  Some  symptoms 
appear  only  under  flight  stresses,  while  others  depend  upon  the  mode  of  equipment  use. 

18 As  noted  in  Sec.  Ill,  such  serial  number  tracking  of  aircraft  is  also  essential  to 
improving  the  capability  of  maintenance  technicians  to  identify  units  of  equipment  (both 
LRUs  and  SRUs)  having  faults  that  are  evading  detection  by  the  standard  tests  used  by 
shop  and  depot  technicians. 
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To  productively  achieve  its  full  potential,  thorough  reporting  of 
symptoms  for  suspected  faults  will  require  a  new  mind  set  for  pilots, 
technicians,  maintenance  supervisors,  and  maintenance  managers. 
Pilots  and  technicians  need  an  environment  where  they  can  feel  com¬ 
fortable  in  documenting  and  discussing  symptoms.  For  example,  rather 
than  penalize  squadrons  and  wings  for  the  quantity  of  symptoms 
reported,  incentives  need  to  be  created  to  encourage  dialogue  between 
pilots  and  maintenance. 

The  end  goal  of  unit  evaluations  should  be  based  on  the  achieved 
condition  of  the  unit’s  equipment  rather  than  intermediate  measures  of 
how  the  unit  achieves  such  an  end  goal.  Indeed,  the  wing  with  the 
highest  rate  of  reported  symptoms  may  have  its  equipment  in  the 
highest  state  of  readiness  in  terms  of  ability  to  deliver  the  full  measure 
of  designed  capabilities.  And  the  wing  with  the  fewest  reported  symp¬ 
toms  could  be  in  the  opposite  position  if  the  reason  for  the  low  report¬ 
ing  rate  is  simply  a  greater  toleration  of  degraded  performance. 

Incentives  must  encourage  a  free  flow  of  such  information  between 
pilots  and  maintenance.  One  way  to  avoid  problems  in  this  respect 
would  be  to  refrain  from  comparing  units  based  upon  their  fault 
removal  efficiency  for  specific  subsystems.  Such  comparisons  could 
make  it  awkward  for  units  to  document  symptoms.  However,  those 
involved  in  the  acquiring  and  improving  of  avionics  should  understand 
which  subsystems  pose  the  greatest  burdens  in  terms  of  fault  removal 
efficiency.  One  way  to  deal  with  such  information  would  be  to  aggre¬ 
gate  it  so  as  to  preclude  the  identification  of  specific  units. 

Application  to  Existing  Subsystems.  An  example  can  show  the 
utility  of  this  indicator.  Suppose  a  subsystem  averages  82  flight  hours 
between  flights  with  a  failure  confirmed  by  the  shop  (MTBF  =  82 
hours)  and  averages  six  flight  hours  between  flights  with  an  indication 
of  one  or  more  faults  (MTBI  =  6  hours).  Although  the  MTBF  indi¬ 
cates  very  good  reliability  for  a  technologically  sophisticated  subsystem 
in  a  modern  combat  aircraft,  the  comparatively  lower  MTBI  raises  a 
flag  about  the  subsystem’s  maintainability,  as  does  the  following  esti¬ 
mate  for  the  fault  removal  efficiency: 

Fault  6  hours 

Removal  =  - —  x  100%  =  7% 

Efficiency  82  hours 

This  result  means  that  maintenance  personnel  could  find  a  shop- 
confirmed  fault  in  this  subsystem  for  only  7  percent  of  the  flights 
where  symptoms  were  indicated  for  one  or  more  faults,  ensuring  that  a 
high  reliability  indicator  does  not  obscure  a  subsystem’s  poor  maintain¬ 
ability. 
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Such  a  low  fault  isolation  efficiency  should  draw  acquisition 
management  attention  to  the  possibility  of  problems  in  one  or  more  of 
such  areas  as  the  timeliness  of  requests  for  maintenance,  the  avionics 
subsystem  itself,  the  BIT,  the  shop  equipment  and  tests,  the  depot 
equipment  and  tests,  the  Technical  Orders,  and  training  of  mainte¬ 
nance  personnel.  To  identify  the  specific  problems  that  are  the  dom¬ 
inant  causes  of  such  low  performance  usually  will  require  the 
subsystem’s  developers  to  field  a  special  engineering  data  collection  and 
analysis  effort. 

Figure  21  shows  how  using  MTBF  and  fault  removal  efficiency  in 
concert  can  provide  a  composite  view  of  the  R&M  of  a  system.  As  the 
figure  shows,  the  F-16  A/B  is  far  better  than  the  F-15  C/D  radar  in 
MTBF,  but  its  fault  removal  efficiency  is  actually  worse.  Nevertheless, 
in  both  cases  the  fault  removal  efficiency  is  still  very  low,  which  can 
actually  confuse  the  difference  between  reliability  and  maintainability, 
especially  when  irrelevant  repairs  occur  in  the  shop.  (Recall  that 
MTBF  is  based  upon  shop-confirmed  failures.)  This  is  one  reason  why 
a  composite  view  (as  in  Fig.  21)  is  needed  to  provide  a  complete  view  of 


failures  Fault  removal  efficiency  l%] 


Fig.  21 — Radar  R&M  goals  for  1990s  fighter  aircraft 


R&M.  Such  a  perspective  would  also  direct  attention  to  the  linkage 
between  the  ability  of  weapon  systems  to  deliver  their  designed  capabil¬ 
ities  and  the  increasingly  critical  roles  of  BIT,  fault  reporting  and 
recording,  shop  test  equipment,  and  depot  test  equipment. 

Application  to  Future  Acquisition  Programs.  Figure  21  shows 
how  fault  removal  efficiency  and  MTBF  could  be  used  to  provide  joint 
guidance  on  R&M  for  the  development  of  a  new  subsystem.  This  fig¬ 
ure  is  similar  to  Figs.  14  and  19,  but  now  the  scale  has  changed  on  the 
vertical  axis  to  make  room  for  more  information.  For  the  1990s,  con¬ 
tractors  are  estimating  that  they  can  provide  a  new  fighter  radar  with 
an  MTBF  in  the  ball  park  of  200  to  400  hours.  At  one  point,  one  lead¬ 
ing  radar  manufacturer  was  estimating  an  MTBF  of  about  400  hours. 
Senior  engineers  for  another  leading  radar  manufacturer  have 
expressed  the  belief  that  an  MTBF  of  about  200  hours  would  be  ambi¬ 
tious,  but  achievable  if  some  new  technologies  mature  rapidly  enough. 
Realistically  and  practically,  estimates  at  this  stage  of  R&D  often 
exceed  eventual  field  experience  by  a  factor  of  two,  sometimes  by  a  fac¬ 
tor  of  ten.  At  this  point,  striving  for  a  goal  of  about  200  hours  (actual 
field  experience)  seems  reasonable  and  is  consistent  with  the  notion  of 
an  overall  quadrupling  of  weapon  system  reliability.19 

In  the  fault  removal  efficiency  dimension,  there  is  little  historical 
information  upon  which  one  can  base  a  goal,  other  than  the  F-15/F-16 
Radar  R&M  Improvement  Programs.  Targets  for  improved  fault 
removal  efficiency  rates,  given  the  investments  identified  in  Sec.  IV, 
are  at  about  the  50  percent  level.  These  are  substantial  improvements 
over  the  7  percent  fault  removal  efficiency  rate  for  the  F-16  radar  and 
the  20  percent  rate  for  the  F-15  radar. 

Because  there  are  always  going  to  be  some  R&M  problems  with  such 
complex  equipment  as  fire  control  radars,  we  need  to  set  realistic  goals. 
Rather  than  ask  for  perfection,  which  cannot  be  achieved,  we  instead 
propose  something  that  appears  technically  reasonable  by  virtue  of  the 
aforementioned  radar  R&M  improvement  programs.  Accordingly,  we 
suggest  that  R&M  management  strive  for  a  combination  of 

•  The  50  percent  fault  removal  efficiency  goal  that  appears 
achievable  for  existing  radars, 

•  A  200  hour  MTBF  (as  eventually  measured  from  field  experi¬ 
ence). 

19The  F-15/F-16  Radar  R&M  Improvement  Program  data  from  1984  yielded  an 
MTBF  of  19  hours  for  the  F-15  C/D  radar  and  82  hours  for  the  simpler  F-16  C/D  radar. 
A  fourfold  improvement  on  these  experiences  would  yield  an  MTBF  in  the  ball  park  of 
76  to  328  hours.  Because  the  sophistication  of  the  next  generation  fighter  radar  will 
probably  be  pushing  the  then  available  state  of  the  art.  as  the  F-15  radar  did  more  so 
than  the  F-16  radar,  a  goal  for  the  next  generation  radar  needs  to  be  cognizant  of  the  F- 
15  radar  experience  in  addition  to  that  of  the  F-16. 
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The  important  aspect  of  this  suggestion  is  not  the  precise  numbers  but 
rather  the  illustration  of  a  process  for  establishing  balanced  goals.  Bal¬ 
ance  is  essential  to  direct  appropriate  attention  and  resources  to  each 
of  the  important  dimensions  of  R&M. 

Accelerating  Maturation  of  Avionics 

This  element  of  the  strategy  for  strengthening  the  acquisition  pro¬ 
cess  aims  at  more  timely  and  fuller  achievement  of  R&M  goals. 
Although  it  focuses  on  aviation  electronics  (avionics)20  because  this 
class  of  equipment  currently  presents  the  greatest  R&M  challenges,  the 
general  concept  of  maturation  is  applicable  generally  to  the  develop¬ 
ment  of  very  complex  systems. 

General  Concept  of  Maturation.  Our  general  concept  of  matura¬ 
tion  is  based  on  viewing  the  research  and  development  of  a  very  com¬ 
plex  weapon  system  as  a  process  that  has  six  basic  phases: 

I.  Technology  development 

II.  Critical  component  development 

III.  Subassembly  development 

IV.  Assembly /unit  development 

V.  Subsystem  development 

VI.  Weapon  system  integration  development.21 

An  orderly  and  efficient  development  program  will  invest  just  the  right 
amount  of  time  and  resources  in  each  phase;  and  while  phases  will 
overlap,  they  will  be  neither  initiated  nor  terminated  too  soon. 

During  each  basic  phase,  we  use  the  concept  of  maturity  as  a  quali¬ 
tative  gauge  of  the  status  of  development  efforts.  We  say  that  a  phase 
has  reached  maturity  when  both  the  state  of  knowledge  and  the  level  of 
performance  (including  R&M  characteristics )  indicate  that  it  is  reason¬ 
able  to  initiate  the  next  phase.  The  term  full  maturity  designates  the 
situation  where  knowledge  and  performance  have  advanced  to  where  it 
is  reasonable  to  terminate  the  development  work  within  that  phase. 

Our  general  concept  of  maturation  holds  that  whether  we  are  decid¬ 
ing  to  initiate  or  to  terminate  a  phase,  the  decision  should  be  based  on 
scientifically  accumulated  evidence  of  progress  and  an  objective 

including  airborne  electronic  warfare  equipment. 

2IIn  the  case  of  an  aircraft  weapon  system,  recall  that  we  define  the  weapon  system  to 
include  the  system-peculiar  ground  support  equipment  in  addition  to  whatever  other  data 
and  equipment  are  required  to  fully  restore  the  capabilities  of  the  airborne  equipment. 
Such  equipment  goes  through  the  development  phases  just  as  does  the  airborne  equip¬ 
ment.  Phase  VI  is  then  responsible  for  integrating  all  of  these  elements  of  the  weapon 
system. 


71 


assessment  of  the  likelihood  that  lingering  difficulties  can  be  resolved 
before  efforts  in  the  next  phase  get  too  far  along.  Within  this  frame¬ 
work,  the  soundness  of  the  basis  for  decisions  can  be  examined  in 
terms  of  the  extent  to  which  phases  preceding  a  decision  have  system¬ 
atically  explored  possibilities  and  alternatives  using  the  scientific 
method  of: 

1.  Observing 

2.  Questioning 

3.  Forming  hypotheses 

4.  Testing  hypotheses  by 

a.  Measuring 

b.  Organizing  and  recording  data 

5.  Interpreting  data 

6.  Drawing  conclusions. 

From  the  vantage  point  of  our  general  concept  of  maturation,  when 
development  efforts  fail  to  fulfill  their  technical  potential,  it  often  can 
be  viewed  as  a  consequence  of  one  or  more  of  three  basic  types  of 
development  problems: 

•  Type  1.  Flawed  application  of  the  scientific  method, 

•  Type  2.  Premature  initiation  of  a  basic  phase  of  development, 

•  Type  3.  Premature  conclusion  of  a  basic  phase  of  development. 

For  avionics,  such  problems  can  be  observed  during  each  basic  phase  of 
development;  and  although  development  programs  may  survive,  R&M 
characteristics  often  suffer.  To  reduce  the  occurrence  and  the  influ¬ 
ence  of  such  R&M-degrading  problems,  RAND’s  current  research  on 
avionics  R&M  has  emphasized  strengthening  the  acquisition  process 
during  during  Phase  V  (subsystem  development)  and  Phase  VI 
(weapon  system  integration  development).22 

Particular  Concept  of  Maturational  Development.  For  these 
phases,  we  have  proposed  a  particular  form  of  the  general  concept  of 
maturation  that  we  call  maturational  development.  Because  the 
development  of  avionics  subsystems  and  associated  support  equipment 
often  proceeds  so  quickly,  little  of  the  systematic  maturation  advocated 
by  the  general  concept  takes  place.  To  correct  this  situation,  we  have 
proposed  that  the  Air  Force  institute  a  formal  period  in  the  acquisition 
process  for  which  development  programs  would  be  required  to  set  aside 
time23  and  resources: 

^Gebman,  Shulman,  and  Batten,  1988. 

23The  acquisition  process  must  resist  overemphasizing  speed  of  development  because 
an  undue  emphasis  on  speed  can  lead  to  failure  to  (1)  collect  accurate  and  relevant  data 
concerning  potential  problems  with  maintaining  the  full  measure  of  designed  perfor- 
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•  Measuring  operational  experience,  organizing  and  recording 
R&M-related  data,  interpreting  the  data,  and  drawing  conclu¬ 
sions  about  the  root  causes  of  the  dominant  R&M  problems 
that  are  responsible  for  any  shortfalls  in  needed  R&M  charac¬ 
teristics.24 

•  Correcting  R&M  deficiencies,  preferably  before  PMRT. 

This  particular  concept  is  based  on  the  thinking  that  a  design  for  a 
very  complex  subsystem  and  its  associated  support  equipment25  should 
be  viewed  much  like  a  set  of  hypotheses  until  rigorous  testing  and 
operational  experience  demonstrate  that  the  design  delivers  the  needed 
levels  of  performance  (including  R&M).  The  prudence  of  such  think¬ 
ing  is  supported  by  the  recurring  observation  that 

Until  they  have  had  detailed  data  about  actual  operational 
experience,  developers  have  failed  to  identify  the  root 
causes  of  the  most  serious  problems  that  have  plagued  their 
equipment’s  operation. 


Design  methods  and  tools  cannot  predict  all  causes  and  modes  of 
degraded  performance.  During  the  design  process,  designers  lack  com¬ 
plete  information  about  new  materials  and  processes  needed  to 
manufacture  avionics  subsystems  with  advanced  technology.  Moreover, 
they  cannot  foresee  all  environments  in  which  the  avionics  subsystems 
will  perform,  nor  all  complex  interrelations  among  them  because  of  the 
the  multiple  functions  of  most  subsystems,  the  multiple  operating 
states  for  many  functions,  and  the  complex  integrations  among  subsys¬ 
tems. 


mance,  and  (2)  redesign  portions  of  the  weapon  system  and  its  support  system  to  avoid 
these  problems.  When  fielding  new  weapon  systems,  we  need  to  watch  not  only  the  time 
to  IOC  but,  more  important,  the  largely  ignored  time  to  a  fully  matured  operational  capa¬ 
bility.  The  latter  can  take  much  longer,  as  can  be  seen  in  the  time  needed  to  introduce 
radar  R&M  improvements  for  the  F-15  and  F-16.  This  maturation  time  would  be  short¬ 
ened  considerably  if  the  Air  Force  scheduled  a  formal  maturation  process  into  the  overall 
acquisition  process. 

^Gebman,  Shulman,  and  Batten,  1988,  refers  to  such  a  set  of  actions  as  Stage  1: 
Data  Collection  and  Analysis.  This  stage  collects  data  based  on  operational  experience  to 
determine  where,  how,  how  often,  and  why  combat-essential  avionics  subsystems  may 
fail  to  deliver  their  full  designed  capability.  Stage  2.  Implementation  of  Further  Develop¬ 
ment  involves  a  further  development  effort  aimed  exclusively  at  reducing  the  frequency 
of  failure  and  degraded  performance  and  at  increasing  maintainability.  Using  the  data 
and  analyses  from  Stage  1,  the  developer  may  have  to  change  materials,  derate  certain 
components,  and  redesign  others.  He  may  also  have  to  impose  tighter  quality  control  on 
certain  materials  and  production  methods.  To  improve  maintainability,  the  developer 
may  have  to  change  the  BIT  circuitry  and  programs,  the  system  partitionings,  and  the 
test  points.  He  may  have  to  add  test  points  and  capture  more  information  about  the 
airplane’s  mode  of  use  at  the  time  of  degraded  performance. 

^Including  software  and  Technical  Orders  governing  its  operation  and  maintenance. 
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Consequently,  designers  need  at  least  one  development  effort, 
including  an  opportunity  to  gather  information  from  field  operations, 
to  understand  the  dominant  R&M  problems  and  their  causes.  The 
government  likewise  needs  such  an  operational  evaluation  to  gauge 
status  and  understand  the  magnitude  of  the  further  development 
efforts  needed  to  achieve  its  R&M  goals.  For  particularly  advanced 
and  complex  equipment,  such  a  pattern  of  iterative  development,  illus¬ 
trated  in  Fig.  22,  may  require  several  efforts  to  achieve  needed  goals. 

Proposed  Applications  for  Maturational  Development.  We 
recommend  a  formal  maturational  development  phase  for  three  classes 
of  complex  combat-essential  avionics  subsystems: 

•  T  ew  subsystems  that  are  just  beginning  development, 

•  Existing  subsystems  that  are  being  modified  to  improve  their 
functional  performance, 

•  Existing  subsystems  where  improvements  in  reliability  and 
maintainability  would  substantially  narrow  the  gap  between 
designed  performance  and  operationally  available  performance. 


•  Develop  a  producible 

design 

•  Operationally  test 

—  Reliability 
—  Fault  isolation 

•  Independently  evaluate 

—  Reliability 
—  Fault  isolation 

•  Does  design  meet 

maturational 

requirements? 

•  Mature  design  ready 

for  integration  into 
host  weapon  system 


(if  necessary) 


Fig.  22— Maturational  development:  A  multiple  phase  process 


Maturational  development  can  achieve  the  largest  benefit-to-cost 
ratio  when  aimed  at  developing  selected  new  avionics  subsystems  that 
are  just  beginning  development  (Phase  V  of  the  development  of  a 
weapon  system).  In  such  cases,  it  should  occur  before  high-rate  produc¬ 
tion  to  avoid  the  high  costs  of  retrofitting  hardware.  For  this  class  of 
equipment,  the  Air  Force  might  most  profitably  begin  with  new  avion¬ 
ics  for  the  new  Advanced  Tactical  Fighter  (ATF). 

A  maturational  development  phase  can  greatly  improve  selected 
existing  avionics  equipment  particularly  when  improvements 

•  Involve  large  changes, 

•  Affect  the  weapon  system’s  overall  design, 

•  Increase  the  airplane’s  modes  of  operation, 

•  Substantially  raise  performance  specifications  for  existing 
mouc-s, 

•  Make  a  large  proportion  of  its  components  no  longer  inter¬ 
changeable  with  those  of  the  previous  model. 

Furthermore,  already  fielded  equipment  can  also  benefit.  In  these 
respects  the  Air  Force  should  consider  instituting  maturational 
development  for 

•  Fire  control  radars  such  as  the  APG  68  for  the  F-16  C/D,  APG 
70  for  the  F-15  E,  APQ  164  for  the  B-1B; 

•  Electronic  countermeasures  equipment  such  as  the  internal 
countermeasures  set  for  the  F-15,  the  threat  warning  system  for 
the  F-16,  the  advanced  self-protection  jamming  system  for  the 
F-16,  and  the  jamming  pods  with  application  to  numerous 
weapon  systems; 

•  Weapon  delivery  systems  for  the  F-15  and  F-16; 

•  Low  altitude  night  targeting  infrared  (LANTIRN)  navigation 
and  targeting  pods. 

Although  each  of  these  programs  has  an  ad  hoc  plan  for  maturing 
R&M,  we  believe  that  each  would  benefit  from  taking  the  further  step 
of  committing  to  a  formalized  process  where  funds,  time,  and  responsi¬ 
bility  would  be  established  in  accordance  with  a  standardized  format 
and  reviewed  accordingly.  Lacking  such  commitment  and  oversight, 
the  time  and  funds  for  R&M  maturation  are  constantly  at  risk  as 
development  programs  suffer  through  the  growing  pains  of  achieving 
needed  levels  of  functional  performance. 
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Reorganizing  Avionics  Development 

This  final  element  of  the  strategy  for  strengthening  the  acquisition 
process  aims  at  reducing  the  R&M-related  development  problems  that 
occur  throughout  the  process  of  developing  a  weapon  system.  To  this 
end,  we  propose  a  reorganization  that  aims  at  two  goals: 

•  Institutionalize  maturational  development  during  Phase  V  (sub¬ 
system  development)  and  Phase  VI  (weapon  system  integration 
development), 

•  Expedite  the  maturation  and  application  of  new  technologies  by 
better  focusing  both  government  and  industry  sponsored  R&D 
during  Phase  II  (critical  component  development)  and  Phase  III 
(subassembly  development). 

Institutionalizing  Maturational  Development.  Ideally,  subsys¬ 
tem  development  (Phase  V)  would  start  far  enough  ahead  of  weapon 
system  integration  development  (Phase  VI)  to  allow  a  maturational 
development  effort  to  be  underway  before  Phase  VI  begins.  Although 
such  a  Phase  V  application  of  maturational  development  would  have  to 
see  the  subsystem  hosted  on  a  different  weapon  system  for  the  gather¬ 
ing  of  operational  experience,26  the  advantage  of  such  long  lead 
development  of  critical  subsystems  is  that  design  improvements  can  be 
incorporated  before  high  rate  production  for  the  new  host  weapon  sys¬ 
tem. 

Currently,  avionics  subsystems  usually  start  full-scale  engineering 
development  (FSED)  last,  even  though  they  are  probably  the  most 
complex  areas  of  the  airplane.  This  is  because  the  avionics  contractors 
and  subcontractors  are  often  not  chosen  until  after  the  prime  contract 
has  been  let.  The  engine,  however,  typically  starts  FSED  many  years 
ahead  of  the  combat  avionics,27  and  the  airframe  anywhere  from  six 
months  to  a  year  before  many  avionics  subcontractors  even  receive 
their  contracts.  For  example,  Fig.  23  shows  the  major  milestones  in 
the  acquisition  and  development  of  the  F-15.  Concept  formulation  for 
the  F-15  began  in  June  1967,  and  FSED  for  the  airframe  began  in 
January  1970.  While  initial  development  for  the  engine  began  roughly 
16  months  before  FSED  for  the  airframe,  FSED  for  the  avionics 

^This  is  an  incomplete  environment  for  identifying  subsystem  integration  problems, 
but  it  does  provide  an  operational  opportunity  to  exercise  the  subsystem  in  an  airborne 
environment  that  is  a  close  approximation  to  the  intended  operational  setting.  More¬ 
over,  with  complex  subsystems,  it  is  beneficial  to  iron  out  most  of  the  subsystem  prob¬ 
lems  before  attempting  integration. 

^Recognizing  the  sophisticated  and  flight-essential  nature  of  turbine  engines,  the  Air 
Force  usually  (1)  contracts  directly  for  their  development;  (2)  starts  this  development 
long  before  that  of  the  airframe;  and  (3)  precedes  this  with  many  years  of  development, 
testing,  and  redevelopment  of  the  engine’s  more  critical  components. 
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FSED 


Fig.  23 — Major  milestones  in  the  acquisition  and  development 

of  the  F-15 


systems  began  considerably  later.  Indeed,  FSED  for  the  inertial  navi¬ 
gation  system  began  roughly  eight  months  later,  and  FSED  for  the  fire 
control  radar  system  began  roughly  11  months  later. 

Because  of  such  tight  scheduling,  avionics  developers  typically  lack 
adequate  time  to  design  and  mature  their  equipment  before  it  is 
delivered  to  operational  squadrons.28  Moreover,  shortly  after  fielding 
the  initial  squadron,  developers  lose  touch  with  many  of  the  opera¬ 
tional  realities  in  which  their  designs  must  survive.  This  means,  for 
instance,  that  they  lack  engineering  data  on  how  temperatures,  vibra¬ 
tions,  and  flight  loads  may  be  punishing  the  more  fragile  parts  of  their 
design.  The  problems  such  programs  have  faced  are  that 


28The  authors,  over  a  period  of  20  years,  have  personally  witnessed  the  development 
and  fielding  of  four  generations  of  avionics  equipment  that  is  now  being  used  by  the 
F-llls,  the  F-15s,  the  F-16s,  and  the  B-1B.  In  every  case,  one  or  more  crucial  avionics 
subsystems  failed  to  meet  the  original  development  schedule  because  the  size  of  the 
necessary  engineering  effort  was  larger  than  possible  within  the  time  allocated.  In  such 
situations,  equipment  was  fielded  before  all  of  the  functional  performance  specifications 
had  been  satisfied.  Subsequent  modifications,  necessary  to  satisfy  the  performance 
needs,  would  alter  the  equipment  configuration  and  delay  the  point  at  which  configura¬ 
tions  had  stabilized  to  a  point  where  R&M  characteristics  could  be  fully  demonstrated 
and  assessed. 
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•  Inadequate  time  has  been  allotted  in  the  product  improvement 
and  acquisition  processes;  and 

•  Insufficient  funding  has  been  provided  to  pay  for  data  collec¬ 
tion,  analysis,  and  implementation  of  improvements. 

Such  patterns,  while  damaging  to  the  R&M  characteristics  of  contem¬ 
porary  avionics  subsystems,  portend  potentially  more  serious  conse¬ 
quences  for  forthcoming  avionics  developments. 

Improve  Focus  of  Developments  During  Phases  II  and  III. 
Reorganization  of  avionics  development  also  needs  to  streamline  efforts 
in  such  areas  as 

•  The  development  of  critical  components  (Phase  II) — for  exam¬ 
ple,  active  elements  for  an  active  phased  array  antenna,29 

•  The  development  of  subassemblies  (Phase  III) — for  example, 
common  modules  for  integrated  communication,  navigation, 
and  IFF  (identify  friend  or  foe)  equipment. 

Current  interests  in  the  development  of  common  avionics  modules  is 
sparked  by  desires  to  reduce 

•  The  need  for  air  base  avionics  shops, 

•  The  amount  of  new  avionics  equipment  that  must  be  developed 
for  each  new  weapon  system. 

This  technology  allows  a  group  of  common  modules  to  be  used  both 
within  a  subsystem  and  across  several  subsystems. 

Common  modules  also  aim  at  developing  smaller  and  cheaper  LRUs, 
which  in  turn  would  negate  the  need  for  an  AIS.  Current  LRUs  are  so 
costly,  removed  so  often,  and  in  such  short  supply  that  each  air  base 
generally  needs  its  own  avionics  shop.  Because  many  LRUs  cost 
between  $100,000  and  $500,000,  the  avionics  shop  uses  large  sets  of  test 
equipment  to  identify  faulty  SRUs  within  these  LRUs.  The  shops  then 
send  these  less  expensive  SRUs  to  the  depot  for  repair,  reducing  the 
time  and  the  value  of  assets  tied  up  in  the  repair  pipeline. 

To  eliminate  this  need  for  avionics  shops  on  air  bases,  the  avionics 
industry  and  various  Air  Force  organizations  are  examining  the  con¬ 
cept  of  more  modular  avionics.30  Rather  than  building  a  radar  with  less 
than  nine  LRUS,  one  would  aim  at  building  a  radar  with  50,  100,  or 
more  modules.  Like  LRUs,  these  modules  could  be  removed  at  the 

®See  Gebman,  Shulman,  and  Batten,  1988,  for  details. 

30These  efforts  include  an  Air  Force  Avionics  Laboratory  program  known  as  PAVE 
PILLAR,  an  Air  Force  Air  Staff  effort  known  as  Modular  Avionics  System  Architecture, 
and  various  industry  efforts  known  as  Line  Replaceable  Modules. 
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flight  line;  like  SRUs,  these  modules  would  be  cheap  and  could  be  sent 
to  the  depot  for  repair. 

Figure  24  illustrates  the  laboratory  development  of  an  advanced 
design  for  a  digital  processor.  It  includes  so  many  modules  that  it 
takes  up  two  boxes.  Each  section  is  a  different  circuit  board  or 
module.  The  shadings  indicate  four  module  types.  Thus,  within  this 
one  design  there  are  several  different  applications  of  each  module  type. 
If  this  kind  of  multiple  application  of  a  single  module  proved  to  be  the 
design  of  the  future,  it  would  increase  incentives  to  invest  in  maturing 
these  units. 

Most  important,  the  improvements  in  modular  avionics  go  beyond 
the  unit  level  shown  in  Fig.  24  to  the  subsystem  level  shown  in  Fig.  25. 
Here  a  set  of  34  different  modules  have  been  used  like  a  set  of  standard 
building  blocks  to  lay  out  designs  for  three  very  different  subsystems: 
(1)  an  army  ground-based  application,  (2)  a  helicopter,  and  (3)  an 
upgrade  for  a  fighter  aircraft.  With  these  kinds  of  multiple  applica¬ 
tions  of  avionics  equipment  to  other  weapon  systems,  the  possibilities 
increase  for  justifying  resources  to  mature  the  modules.  The  potential 


Fig.  24 — Example  of  a  digital  processor  designed  with 
standard  modules 
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Fig.  25 — Three  designs  built  for  communication,  navigation,  and 
identification  equipment  from  one  standard  set  of  modules 


broad  applicability  of  the  ten  most  utilized  types  of  modules  from  the 
set  of  34  is  illustrated  in  Fig.  25.  Subsystem  number  1  would  need  63 
modules  from  this  top  ten  set.  Subsystem  number  2  would  use  34  from 
this  top  ten  set.  Well  over  half  of  each  of  the  three  subsystems  would 
be  built  from  modules  from  this  top  ten  set.  This  top  ten  set  clearly 
presents  a  very  broad  opportunity  for  benefiting  from  investments  in 
R&M  maturation,  as  do  the  24  remaining  modules. 

While  the  adoption  of  common  modules  has  many  attractive 
features,  it  also  presents  some  R&M-related  challenges  for  the  acquisi¬ 
tion  process.  Administrative  challenges  include 

•  Establishing  a  modular  avionics  architecture  that  has  sufficient 
flexibility  to  support  high  levels  of  maintainability, 

•  Orchestrating  adequate  time  and  resources  to  mature  R&M 
characteristics. 

Technical  challenges  include 

•  Designing  sufficient  fault  isolation  capabilities  into  individual 
modules  and  integrated  sets  of  modules, 

•  Designing  interconnections  between  modules  that  are  reliable 
and  easy  to  fault  isolate,  even  though  the  number  of 
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interconnections  between  modules  seems  to  be  increasing  with 
greater  miniaturization  of  electronics. 


SUMMARY 

The  major  weaknesses  in  the  acquisition  process  are  the  absence  of 
meaningful  measures  of  R&M,  such  as  an  indicator  of  fault  removal 
efficiency,  lack  of  a  process  for  setting  rationally  based  and  balanced 
R&M  goals  for  future  equipment,  and  weak  assurances  that  needed  lev¬ 
els  of  R&M  will  be  delivered. 

Strengthening  the  acquisition  process  to  deal  with  these  weaknesses 
must  start  by  expanding  designer  awareness  of  R&M  deficiencies, 
especially  of  problems  with  current  avionics  equipment.  Although 
there  is  a  lot  of  emphasis  on  reliability  and  some  people  believe  that 
levels  of  2000  hours  MTBF  are  attainable,  there  is  no  comparable 
enthusiasm  for  maintainability.  This  study  suggests  fault  removal  effi¬ 
ciency  as  a  way  to  stimulate  such  emphasis.  Maturational  development 
is  one  way  to  assure  delivery  of  specified  levels  of  R&M  on  complex 
avionics  equipment. 

Strengthening  the  acquisition  process  also  calls  for  accelerating  the 
maturation  of  avionics  equipment  currently  in  the  field.  Because 
future  avionics  will  account  for  40  percent  or  more  of  the  fly  away 
costs  for  future  fighters,  more  attention  must  be  paid  to  maturing  it 
quickly  to  avoid  high  costs  of  spares  and  retrofitting  improvements. 

Finally,  all  the  above  strategies  may  depend  ultimately  on  reorganiz¬ 
ing  avionics  development.  This  may  prove  necessary  to  begin  matura¬ 
tional  development’s  first  stage  (data  collection  and  analysis)  early  in 
the  development  of  new  subsystems.  It  may  also  be  necessary  to 
timely  implementation  of  the  second  stage  (implementation  of  further 
developments).  By  scheduling  a  formal  maturational  development  pro¬ 
cess  into  the  development  of  complex  subsystems,  and  by  providing  the 
necessary  funds  and  time  to  accomplish  maturation,  the  Air  Force  can 
go  a  long  way  toward  assuring  that  needed  R&M  levels  are  actually 
delivered. 


VI.  RECOMMENDATIONS  AND  CONCLUSION 


Although  the  traditional  view  of  R&M  may  deal  adequately  with 
problems  that  are  consistent  show  stoppers,  a  new  view  of  R&M  is 
needed  to  deal  effectively  with  faults  that  only  degrade  performance 
and/or  manifest  situation-dependent  symptoms.  Because  many  of  the 
R&M  problems  with  modern  avionics  are  of  the  latter  category,  we 
have  proposed  a  new  approach  to  viewing  R&M  that  emphasizes  the 
two  most  fundamental  R&M  characteristics: 

•  Fault  initiation  and 

•  Fault  removal. 

Managers  need  a  minimum  set  of  indicators  that  provide  a 
comprehensive  characterization  of  how  a  system  or  subsystem  is  doing 
in  terms  of  these  fundamental  characteristics.  We  have  suggested  two 
parameters  for  such  purposes: 

•  Mean  flight  time  between  flights  with  shop  confirmed  failures 
(MTBF)  and 

•  Fault  removal  efficiency. 

Using  such  parameters  to  analyze  contemporary  field  experience,  we 
find  progress  in  MTBF  not  being  matched  by  progress  in  fault  removal 
efficiency.  We  have  proposed  several  actions  that  aim  at  bolstering  the 
capability  of  the  support  process.  Such  improvement  alone,  however, 
will  not  be  sufficient.  Product  improvement  actions  must  examine  the 
dominant  causes  of  low  fault  removal  efficiency  and  important 
improvements  must  be  expedited  to  bolster  the  supportability  of 
specific  equipment.  The  high  expense  of  such  product  improvement 
makes  it  far  more  desirable  for  future  acquisition  programs  to  build  in 
high  fault  removal  efficiency  capabilities. 

Strengthening  the  acquisition  process  alone  is  not  sufficient  in  the 
near  term  because  of  the  low  rate  of  turnover  in  military  equipment. 
Even  with  the  best  of  improvements  in  acquisition  programs,  there  will 
still  be  major  needs  for  bolstering  product  improvement  programs  and 
strengthening  the  support  process  to  cope  with  problems  currently  in 
the  field.  For  these  reasons,  the  last  three  sections  have  developed  a 
strategy  for  strengthening  the  Air  Force’s  capability  to  manage  weapon 
system  R&M  during  all  phases  of  the  weapon  system  life  cycle. 
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Perhaps  most  obvious  in  the  strategy  summarized  in  Fig  26  is  that 
at  each  stage — not  just  support,  but  also  product  improvement  and 
acquisition— there  is  a  critical  need  for  better  information  from  the 
field.  Maintenance  personnel  need  more  complete  information  to  deal 
quickly  and  effectively  with  faulty  assets  that  escape  repair,  especially 
with  the  less  than  fully  mature  equipment  already  in  the  field  today. 

To  improve  the  isolation  and  correction  of  faults  that  degrade  per¬ 
formance,  the  Air  Force  needs  to  improve  the  quality  of  information 
received  from  the  pilot  debrief  and  improve  the  tracking  and  correction 
of  R&M  deficiencies.  When  equipment  resisting  repair  efforts  is 
located,  the  support  process  needs  greater  capabilities  to  fix  hard  prob¬ 
lems  so  that  faulty  equipment  does  not  circulate  between  shops  and 
aircraft  in  degraded  condition. 


Support  process 


Product  improvement 


Acquisition  process 


Fig.  26 — A  cohesive  strategy  for  improving  R&M 


For  enhancing  the  product  improvement  effort,  we  recommend 
improvements  in  the  systems  for  gathering  information  and  engineer¬ 
ing  data  about  field  problems,  an  increase  in  the  amount  of  field 
engineering  analysis,  and  expediting  of  important  improvements  in 
products. 

For  improving  the  acquisition  process  we  recommend  expansion  ot 
the  awareness  of  deficiencies  in  the  process,  adoption  of  the  maintain¬ 
ability  indicator  of  fault  removal  efficiency  and  of  maturational 
development  to  help  mature  new  avionics  equipment,  and  reorganiza¬ 
tion  of  avionics  development  to  better  address  current  problems  and  to 
meet  future  challenges. 
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